Advogato blog for tampe

6 Oct 2008

Mon, 6 Oct 2008 18:02:11 GMT

Some ideas just fly, where?, I don't now but it's butyfly

Using backtracking to parse is sort of inefficient, you take one try at the time. Using the standard tools for parsing (bison yacc lex) is way faster, so, why am I going to choose to use backtracking?

Easy mental model, compact code, extensible. The idea is that users and library writer should have more possibilities to add good checks and extend the parser - at least I have many times missed these features. The hope is that the approach I'm using will make this not only possible but also not to hard to do

But performance, what about performance? Well think about how we code, we have one big code base and incrementally line by line modify it a little. Why not store extra information so that the parser only backtracks on new code sections by using the extra information to select the correct branch directly (when order of choices does not matter)- is that enough? I don't now I'm pretty new in this game. You can if you like do analysis as well to align the choices in order of likelihood and tricks like that. Maybe I'm wrong but I don't expect this to be an issue practically when taking this observation into account. But things can add up, backtracking, lisp instead of C++, flexible but slow data structures. No big deal, I can as well backtrack if my code can.

I will now dive into code again, but one thing I'm missing in the picture is the buffer pattern, this is a thing that is really nice to have, why not extend that to work in a tree context, think about that. A lot of nice butterfideas will come out of that mental exercise, colorize me, colorize life, colorize you

Cheers to you all

5 Oct 2008

Sun, 5 Oct 2008 22:20:22 GMT

Arborists unite

Have been sick last week, nothing much have been done most of the parser loop example or my endavour in prototyping and experimenting with generators about trees is finished though. Must of this stuff was just a simple generalization of previous stuff so 98% of the stuff should be finished. The logic should needs testing. So whats new ideas have come up of prototyping?

Well, for one thing catching and modulating signals and their values has to be streamlined into the framework.

To have some more notion of setf of generator stuff may be of value (think of letting A represents an element in a data-structure, you might want to set that value to something). this is not anything functional minded persons would recomend, but locally in a transient that functionality is nice to have in order to reach efficiency.

managing state of collectors, transformers and diferent constructs is needed or simplifies logic a lot. This is implemented and the framework should now allow for inclusion of similar constructs that adheres to the same pattern .

Noticed that applying a structure of transformers to a similar structure of transformers should be possible think of making sense of


  (apply [Tr1 [Tr2 Tr3] [val1 Tr4])

Lot's of ideas is maturing in handling of parsing and some of it is proturberized into an examplification.

If you want to check out some of this code with terse comments see loop3 , you can check out my first tree effort in loop1

If you are going to read this stuff, happy sleep and good night

26 Sep 2008

Fri, 26 Sep 2008 17:45:31 GMT

Tracking Signals

Prototyping using the framework for parser generators touches one more aspect. The aspect of needing to walk down the tree multiple times when backtracking. If what you have is an abstract generator X, you will want to first try pattern M1 against X, then restart X and try say pattern M2 against X. Enter the need to push and pop the state of X. Actually you could think on many more of these operations like storing a state in variable A, printing the state, make the loop persist and so on. All these operation and similar operations has a common pattern, so I made macros and now introduce these kind of pattern will more easally done.

The pattern is this. A generator has state. And depends on other generators, thats it. The dependency of all generators will be represented by an acyclic graph. Now in order to make the default illusion you will have to make sure at many places that you will update and therefore the signals hit each object multiple times. This is ok because the logic is to first send a clear signal that on all objects with internal state set a flag to true. then when the update signal such as next push or pop reaches the object it will if the flag is true do the action and then set the flag to false. This makes the design clean but costly. Also the model of the action of a next for example will trigger a flow of signals backwards up the call graph in order to skip finish and yield correctly. The next is also special because also updates internal state by using the internal state and the dependents state

There will also be a flow of data as well, and that is the other category of flow and can be represented by the value av the generator or a view of the state of the generator. There is three main abstract streams, ordinary data, delayed data and lazy data. ordinary data is data as you normally would consider as data, delayed data is data that is not yet computed, but we treat it as a computed data but all functional calculation based on that data will be delayed as well. Lazy data is a lambda that when there is a need for that data, the it will be calculated by evaluating the function.

So The framework is dead slow, but I do not want to make a compiler just yet. The idea is to write a slow but full featured loop environment, have nice syntax. Iron out and streamline the code, construct an extensive set of test cases, then make a compiler using this framework (eat the dog-food) include the forward chaining stuff as well, this beautifies a lot of logic I hope.

What's next after parsing in protyping?, well one thing I'm thinking of is tree manipulation. Consider having a tree, to do a macroexpansion in a lisp tree can be viewed as a way to have a walker walking the tree and when there is a match, transform things that comes "after" and "below" on the same level. Now you might want run many of these modifiers one after the other, (to reuse L2 cashes if multiple cores and L1 cashes if they are on the same core) This is an interesting pattern and will brake the functional approach so it would be interesting to try to design some framework to handle this. Again it will be a nice exercise.

I hope that you see the unix in all this and that you can view it as a nice extension of the piping idea, or as a generator for extensions of the piping idea

Have fun!

23 Sep 2008

Tue, 23 Sep 2008 20:52:35 GMT

Lock out for his eyes, or else you will meet the Bug God

I'm coming from mathematics and to do my Phd I mainly stared at an equation on a piece of paper most of my time. Thats almost 4 years of staring. Oh boy, it was difficult. Oh boy it was exciting. I almost lost my mind. Anyhow nowadays I try to spend time on less difficult things and surf on my achieved abilities. Usually that's what's more practical oriented work means don't be too smart, keep it simple.

I've now spend one week of staring, trying to find a bug in my loop code. I removed it 15min ago and now all the stuff I had on my web-page referred to in the previous post is coded. Why is it so hard? Maybe I'm not good enough to do this. This is very frustrating, I mean I know that I have something powerful in mind, I run around in all direction expanding the prototyping. Why is boilerplate code so easy to write and an real implementation so far ahead? - It's complex to get the all the details correct, that's it. Your mind just sees the main stream

Ok, what's next? Well let's see a goal of mine. I want to find the ultimate way of handling and analyzing code. And doing the analysis should be focused towards a dynamic environment more like Qi,lisp,emacs,python,ruby,perl... You could still code C, C++, but your coding environment should be dynamic and modular like unix. Now what's are the elements doing this analysis. Walking a code tree - that's more or less trivial. But the path I indicated on the previous referred web page can be extended to include top-down parsing as well. The idea is that we need code analysis tools that are easy to extend and modulize more than we need a fast one. Therefore the use cases of top-down parsing is going to be an important benchmark for the loop environment. ---- Let's open up the closed loops inside your compiler, Let's prototypoize a new web page.

By the way, the forward chain stuff is also converging. In one month or so I will be able have something that I'm satisfied with.

Good night

18 Sep 2008

Thu, 18 Sep 2008 20:53:12 GMT

Wake up, the world is turning

Oh gosh, I've lost my track and spend most of my spare time exchanging my home computer for a new one. Some things went smoothly somethings took days to decide on. Anyway I'm up and running again.

A Couple of times a year I usually go through times where I'm being very tired. It's probably my body that just need to rest and recharge. I think that it's important to listen to these signals and really take time off to sleep more and not spend so much time in thinking mode.

I've planned to get more of my loop stuff finished prototyped, but it takes time. On the other hand this stuff is really fun and powerful (me thinks at least) and I wanted to try explaining why I find this so interesting, so I compiled a page for any interested reader to digest, (I'm trying to render the code a little nicer, used stuff from planet lisp)

Have fun, Cheers!

7 Sep 2008

Sun, 7 Sep 2008 13:36:53 GMT

Keep the pokers isolated

By restricting your toolbox you can do more. This is so true in my work with my iterate frame work. By assuming that you use the framework in a functional manner, and yuo should unless there is good reason not to, you can do fancy things with this restricted framework.

What I'm learn't however is to take advantage of the more common imperative methods and constrict them into well defined small constructing units and then when combined into a framework transform the functial lang to a new much richer functional languish.

As an example, if you do a very advanced looping, millions of times and you enters a error, if you use the framework I'm designing it should be easy to store the state of the loop 5-10 steps back so that at an error, you could rollback to a previous stored state and then do the iteration again step by step and examine the state to do the analysis of how an error appeared.

CTRL-W in firefox you emacs fan-boy, argh and cheers

3 Sep 2008

Wed, 3 Sep 2008 19:01:28 GMT

Timetravel

One of the things I design and code around is the ability of time travel, really! This is the feature that most don't need but could be cool to have. So how could you implement this. The model example is the case when you have to peek into the future of an in-coming sequence and then return to now and continue the iteration. There is two ways I can see of doing this, one is to clone the state of iteration or push it and continue with the cloned one to look into the future or pop the stack afterwards. The other one is to return a place holder and fill in the value when you have it and do the magic unwinding by applying the unwinding logic yielding a true value in the end. I think that both methods could have it's place but I planned to implement only the straightforward clone or stack approach thinking that the other one is too exotic. Wrong I was, Being able to use placeholders gives a much more natural code in parts of my loop macro handling tree iteration logic and should actually be an effective pattern there. So I implemented that, 70 lines of extra code.

31 Aug 2008

Sun, 31 Aug 2008 19:49:17 GMT

How can something so simple be so interesting

The basic of programming, wrap things up in a loop or do a recursion and you will touching one of the cornerstones of computers that makes them so much more effective than humans in many problem spaces. So I am amazed that there is new things to learn at least for me. The thing I'm doing is implementing the generator framework you can see in python and marry it with the ideas coming from frameworks like the loop macro in lisp and the iterate loop framework for lisp (google for lisp and iterate, I found the documentation and links really interesting at their site)

So what is so cool with this task? - Well the thing is that a lot of basic patterns come alive in front of you, and it's really interesting and fun to capture them and feed them to the framework. It's sort of a pure mathematical feeling

So I've been busy coding, exploring, debugging and designing. Hope you can enjoy this as well in a few weeks time, until then

Happy Hacking

18 Aug 2008

Mon, 18 Aug 2008 20:32:48 GMT

Not too close please!

Thinking about how the multiple core setup is, I realized I missed something in my internal map how computing is done on my machine, say 2 cores. Many applications have good cache hit rates but it strikes me that when you do scanning work in a stream with a sequence of modifiers you would like to have the stream nodes close to each other in order to reuse cache but not too close in order to muck with each other. For large mungling of data sure there will be a benifit on doing this.

Another thing that struck me was that perhaps working with variable sized objects might not be as stupid as I thought before, especially if your work flow is to mostly construct new lists all the time. Then you could compress your linked list for the special case when the cons cells is physically situated one after another or perhaps just close to each other. Hmm actually you could make your cpu help you in this task to really scan fast through these lists. And perhaps there are algorithm that may still let you link as you like (at the usual performance hit if it is done more often). Think scanning speed of arrays, flexibility of linked lists.

Actually contemplating about scanning and reading in a stream from RAM, doing processing on the cores in a stream like fashion you would like to use the bandwidth of the BUS between ram and cpu as efficiently as possible, can it be wise to implement help in the cpu to compress and decompress data structures?

Bullseye!!!

17 Aug 2008

Sun, 17 Aug 2008 20:49:18 GMT

Prototurb that fabulous Code of yours

tree-grep "^my*" code.prt | v-sed s/old/new/ | commit-to -m "Change old to new" code.prt

in tree-grep, ^ means the beginning of a list representing a node in the tree

v-sed = keeps version data, and stores the changes in the default work branch,

commit-to = commits the work-branch

Nothing new to deliver, just having this protoplasmic brain that produces some ectoplastic thoughts eg. just alien ideas to make our thoughts phone home. And you, those spooky cryptic comments i have is nothing to get scare about, it's all about creativity and the fun it represent.

I've not been lazy, but I'm searching for the toolbox and the format of how I would like to do development. It has for the 6 past months actually been a constant flux of creativity and I learned a lot. I hope that I can start to get ready to knit together at least prototypes so that others can share the fun.

Let's recycle the proto stuff, I mention leo,git,python and Lisp in the previous post and prototyped a little about the git,lisp and python association in the beginning, what about leo, the leo maintainer arguments that when developing and fixing a Bug It is nice to collect the bit's that are associated with the Bug and work on it in that environment. Now, you might disslike that method but it does something interesting, it documents very effectively the Bug without doing any comments at all and will probably help the next person fixing the Bugs that the referred bugfix will introduce. I'm not eating this dogfood though but I can imagine it beeing useful so I included it.

The problems with associations of cause is that you don't transmit what you see and it's not exact, but It do let people think for themselves and the intention is of cause to be overrun with better ideas or arguments.

And to be exact all this is free thoughts about what our environment would look like if there was a imposed structure on our files and what that structure might bring and look like

Let's think, some mentioned gnu screen together with git and file systems, cool. I associate gnu screen with a state that you can hook into, work (with/in) that state, and than log out. This interpreted in a datastructure could be that you are working for example with different branches of the datastructure in different screen sessions, e.g. what you do in one screen session should not interfere with what you do in another screen session until you do a merge if that is what you wan't. if you do not merge but quits a GC perhaps needs to kick in and clean the stuff up.

It all boils down, in the end, to one thing, A Smile )