Sunday, September 30, 2012

Dart Koans

The next week or so, I'm going to be limited in the amount of time I will have for some programming. I will hopefully still get some programming in but I will be limited in how much time and frequency I will have for it. Because of that, and the fact that I'm a little impatient, I've decided to share some information on my current project.

The project is Dart Koans. It is inspired, though not based on, a ruby project of a similar name: Ruby Koans. I completed the Ruby Koans when I was first learning Ruby, having come from other similar languages like Python as well as from JavaScript. The Koans helped a lot because I learn really well from hands-on coding. Additionally, while covering the basics, it does dive right in.

I felt that a similar approach to learning Dart would be a great contribution to the community. There are many programmers coming in from other languages who are comfortable with the basic concepts and don't need some of the background or theory involved with traditional tutorials. They're familiar with what a variable is, and just want to know how they're used in Dart. They're itching to write some code in Dart to see how it feels.

The Dart Koans are for people new to Dart, and comfortable on the CLI (currently it's not recommended to run from the Dart editor, as it's blocked by a couple of bugs.
1) Issue 4654 (Dart Editor should handle ANSI color output).
or
2) Issue 2789 (Provide a way to find out if stdout is connected to a terminal).

The idea behind Dart Koans is that you achieve enlightenment through failure. You start with many failing tests. And individually correct each test and re-run the program. With each run of the application you should get closer and closer to resolving all tests. Each test introduces a new concept of the Dart language or libraries. At the moment it is still very very early and only a small portion of the total tests are completed. Currently there are 55 tests. By the time this is completed I expect well over 300 or more. As each test fails it will provide the error, line and file to look at to correct the issue. Each test (using the unittest library) is accompanied by several comments which will provide information about the concept being introduced.

Due to the early development stage, the tutorial is currently very much in the preview stage and far from its completed form. However I intend to add more tests and areas as much as I can. I'm interested in any feedback or comments or concerns but please keep in mind things are very much subject to change. Current output is primarily for demonstration purposes and is most likely not the final result.

Because I will be updating very frequently I will not be making many blog posts about the projecct unless its regarding a significant milestone or involves any major changes to the project.

Wednesday, September 19, 2012

Resort to sort

So the past few days I've been playing around with some simple algorithms. I'm not 'professionally' trained, and most of my experience has been with what I've seen in code or have implemented myself. Does that make me less of a developer for not being classically trained? Maybe, maybe not.

In particular I've been playing around with some of the sorting algorithms. Stuff like Bubble sort, Selection sort, Insertion sort, and of course Quicksort.

My first task as been to study the algorithm and look at whatever pseudo-code or explanation is presented for the algorithm. Study and understand it. Then is to walk away from any computer or anything, take a little break. After a short time I then pull out a pen and paper and write down a Dart version of the code. I write it out in front of me without referring to APIs, or without referencing the material.

I later then enter that code (or re-write it) into the Dart editor in a simple test file I have. I run it, fix my bugs, and run it again. My test file contains all of the various sorts as I learn them, it then prints out the sorted list and the time it took to complete (using a Stopwatch). I also have a function to create a list of X size filed with values from Random.nextInt.

Being able to easily create a random list of a determined size is nice, as it allows me to easily increment the list size as I choose and see the differences in performance. For instance it took a list of almost 5000 items for Quicksort with the two isolates to be faster than the main-thread version of Quicksort. (See Edit note below)

Many of the algorithms I've implemented are without performance improvements and are far from the most efficient versions available. One thing I've noticed though is List.sort((a, b) => a - b); has generally been one of the best performing algorithms, in lists of up to 10,000 items.

However as I mentioned none of the algorithms has been tuned for best performance or memory usage and are just the 'simple' versions of the algorithms just for my own learning purposes and less for actual performance comparisons. The performance comparisons are more of a visual reference to the Big O Notation for myself.

[Edit:]So I was wrong. I had poorly implemented my Quicksort with Isolates. Very poorly. It was a hodgepodge of Futures and a custom callback. I went through and cleaned it up and changed it so it returned a Future instead of calling a callback. Once I did that I saw an over 10x speed increase in the Quicksort with Isolates version and it and was about the same speed as the Main-threaded Quicksort for lists below 500 items. And faster for lists greater than 500 items. This has more to do with my unfamiliarity with Isolates than with the algorithms themselves.[/Edit]

For those who may be interested, you can the current version of the file here: https://github.com/butlermatt/sorting_dart. As time goes by, I will be adding more algorithms (with comments) as I learn them myself. Again please recall these are not optimized in any way and are only presented as-is, as proof of concept.
In time I may add some optimized versions of algorithms myself as I progress. Keep in mind however this is primarily a learning project and not designed for use. They may be (and are) buggy.

Monday, September 17, 2012

Contributed Contributions

So it has been quite some time since my last post. Today will be a relatively short one, but I think I'm going to try and get into the habit of writing a blog post now and then.

Today I wanted to mention a little about contributing back to Dart. This is something very easy for anyone to do, on a variety of different levels. First and foremost, I've been contributing in small ways in the past just by using Dart. And in particular by using Dart and filing bug reports when you run into something. Currently I have 15 bug reports that I have reported and are still open. And many more which have been fixed for a long time.

You can submit bug reports for not only code that isn't working correctly, but also for documentation which may need to be updated or corrected; for updates to the website; or even just style recommendations (particularly those which may apply to the Style Guide).

Bug reports are a great way to contribute as the more bugs that are found and squashed the more stable and attractive Dart becomes.

Eventually you may start feeling comfortable enough to provide solutions to the bugs you provide. Either pasting in a suggestion into the bug report, or by submitting a patch.

Note: I will mention this here ahead of time. Before Google can integrate any code you provide them, you must sign the Contributor License Agreement (CLA).

I have personally submitted patches to a number of areas of the Dart project. My first was to the dartlang.org website. I found a typo in the language tour. So I went to GitHub site. Found the Dartlang/dartlang.org repository and forked it. I made the edits that I saw needed to be corrected and issued a pull request.

One handy tip I received, was to use a rebase workflow when dealing with edits to the github repositories. After doing a bit of searching around, a very good article I found was A Git Workflow for Agile Teams. This shows a very straight forward method of using rebase to create nice clean commits when pushing upstream. I still refer to it when working on patch for the sites.

After submitting a couple of corrections to the dartlang.org site(s) I began to feel comfortable enough with submitting code to the dart source code. This process is significantly different than just forking a github repository. First instead of using github, its best to follow the steps here for getting the source. Make sure you can build before trying to edit any code.

Once you've verified you can build after getting the source code for the first time. You can go ahead and make your changes. After making your changes, make sure you once again build. Then also make sure you can pass the tests. If possible run as many tests as you can, not just specific to one area as you never know what else might rely on behaviour you didn't expect (I found this out the hard way myself).

If everything looks good with the test. Submit your patch. I'll give you a hint now as well. Once the patch is uploaded, it will display a chromium code review site. Similar to: https://chromiumcodereview.appspot.com/10918056/ Be sure to go to that site. Edit the issue and add a reviewer (if you're not sure who should review ask in either the bug report that you're working on or from the mailing list). Update the issue once you've added a reviewer. Then click on 'Publish+Mail Comments'. If you don't do this, no one will know that your patch is waiting for review!!! (Mine waited over a week because I didn't do this haha).

Once your patch is looked at, you will usually get a 'lgtm' (Looks good to me) and possibly some suggestions on small typos to fix. If you get line comments on things to fix, then fix them in your code, then re-upload a patch (using the same method as before). It will automatically associate it with your previous patch (as long as you're using the same git branch!). Then go to the comments in the source that were left for you, click on each. Then click on the 'Done' link. That will signal to the reviewer that you looked at each comment and addressed the issue. Once you've marked each comment as 'Done', then 'Publish+Mail Comments' again.

Usually by this point the reviewer will commit your patch for you into the source. They will usually link another patch ID, and on that patch ID will be a revision id such as https://code.google.com/p/dart/source/detail?r=12278. Once you have that you can also watch the buildbot progress for your patch to see if you end up breaking anything ;)

So that's a summary of how I have, and you can, contribute to Dart. Now get out there and enjoy it!

Friday, June 22, 2012

Abstraction? Encapsulation? Integration?

So, I'm having a bit of a design dilemma with one of my projects. I have a class that represents a switch (physical switch that is). It contains information about the switch such as system description, uptime, firmware versions, etc.
I want to have a list of these switches represented by a number of divs. When programming in PHP, I would make a for loop and just loop through the list of the switches pulling the data into a template-esque string which would be echoed out.
That method isn't very dart-y. I could do it that way but it doesn't feel like the dart way. I would also rather input into the dom and add event handlers more dynamically. However my current solution adds a 'DivElement' to the switch class. I can then template the innerHTML for the div, and add an onClick event handler right into the constructor. Then as my 'manager' class loads the switches it can then append them to the dom with insertAdjacentElement('beforeend', aSwitch.element); to the correct spot.
I like this method for the most part as it allows me to provide the default display information directly in each object. But my problem is that having done at least a little ruby on rails (and other ruby mvc framework programming) I know having my model and my view coupled together like this is a big no no.
But for the small project I have I'm not sure if I really need to worry about adhering too strictly to that methodology.
So I'm at conflict with myself.

Wednesday, May 30, 2012

Long Overdue

Okay wow..... Uhm.. yeah. What can I say? I've been extremely lax in keeping content on here. For a period of time I got pulled into other projects. However recently I was able to work on a project which I was able to use dart, and not just my usual server side work, or internal server but something that would be customer facing. Albeit something very minimal and not actually 'interactive' for the customers who would see it. But still it was a great chance for me to make use of dart and it was nice doing something publicly visible. However due to the nature of it I was unable to share the code for it.

I've also pushed out the source to DartMud as I've not worked on that in some time. I may be able to get some more commenting and eventually some additional work on it, however at the time being it will probably be pretty stagnant for the next little while at least. If you're interested you can check out my git repository.

There have been many many chances to the dart editor, dart vm, et al. that I have no real possibility of listing them all here now. However work is progressing rapidly in all areas and the integrated build is seeing updates almost weekly. It's very exciting to see the changes and additions being added so frequently. And so far only the rare 'breaking change' has come through which requires significant updating to any code. One such change was an update to the way callbacks are assigned on sockets. But it was a minor effort to update DartMud to reflect those changes.

At the moment, my current tests have me playing with the webserving capabilities of Dart. I have just recently begun to play with writing my own server. For those looking to get into it a little faster however, there are a couple of nice projects already in place including an Apache mod_dart, to run dart server side scripts as you would run a PHP script right inside of Apache. Additionally there is Dart Start, which is a (Ruby) Sinatra inspired dart server. In particular Start allows for simple routing and parameter passing as popularized by Ruby On Rails. So by all means check them out! In the mean time I'll continue playing with my server and let you know as things progress.

Thursday, March 8, 2012

Darting here and there

So another Dart Editor update, now at Build 5104, get it here!. I love how quickly the dart integrated builds have been coming lately. This new version does away with the old 'Libraries' view panel on the left side (by default) and instead implements a Files view. Instead of opening a project you now open the folder containing your project. All of your .dart files are loaded into that. I wasn't sure about it at first, as it is a changes in how it would work previously but already I'm seeing the benefits of it. It does help to encourage a Good folder/subfolder layout, which I was already comfortable with anyways.

I've seen a few people report some difficulties loading their existing projects, however I can't really comment on that as I have not encountered those issues myself. This version did correct the mass of errors I was receiving in the previous version regarding variables hiding other variables or methods. However it did introduce a couple of new ones.

First, this version introduced the changes to the dart:io library, changing all eventHandlers over to be in the form of onEvent, and one-shot methods (such as File.exists) now take the callback function as an argument, as opposed to having to specify an 'existsHandler'. As such I had to go through my code and made updates to the Sockets and Files to properly reflect the changes to the library.

Secondly, I ran into an issue where any print statements, within a class, would generate an error or warning that "print" is not a method in , where ClassName is whatever class happened to contain the print statement. This post in the Dart Group provided an easy, temporary fix, that is to simply import 'dart:builtin' where required. A simple fix and helped to clean up the errors there so I could focus on my own errors that I had to worry about.

I notice that while the IDE and SDK were updated to Build 5104, the Dartium Build remains at 5070. At this time I am still unable to get the Dartium builds to run on my machine due to a version problem with the shared libraries. I receive the following error:

error while loading shared libraries: libgconf-2.so.4: wrong ELF class: ELFCLASS64

I believe this error is related to the fact that I'm running a 64-bit OS, but it's trying to load 32-bit build. This may also have something to do with my early ventures in trying to build DartVM, IDE, and Dartium myself and installing the extra libraries, etc, which may have conflicted with what Dartium is trying to load. In fact, now that I think of it, I should run a good apt-get autoclean and autoremove regardless. I don't believe it will fix it, but certainly clean up my system a little at least. Now I've not been too worried about this as my current project doesn't use or require any client-side work, but eventually it might be interesting to have a working copy of Dartium for when I start working on some client side projects. Hopefully the 64-Bit builds of Dartium will be available by then.

Monday, March 5, 2012

Dart Mud... Progress 2

So I decided it's time for another progress report on my Dart based MUD. This project has be a lot of fun, and giving me experience with the dart:io libraries including Files, ServerSockets, and Sockets. Before I get into too much, I will mention that there is a new Integrated Dart Build available. Version 4760 (though SDK appears to be build number 4759). You can find your download here. Since using this build I have been getting errors and warnings when using the dart:io library. However it doesn't prevent my project from running properly. Just distracting thinking I have an error when I don't.

To follow up on my previous post, I have begun using git locally to archive changes and keep a bare-bones changelog system. Already I see this being helpful in keeping track of what I have changed or done, and additionally, in being able to revert changes easily. Once I clean up some of the code and add a plethora of comments, I may pop the project up onto Github as well.

My User Object now allows for the creation of a new character, automatic saving of a user and loading user (and password) from a save file. To save character information I'm currently placing the information I want saved into a map, and using saving it as a JSON object. In the future I can see some issues I may need to work out, but since I haven't decided on how things such as Objects, equipment etc will be implemented yet I don't know yet how much of an issue they will be.

As part of adding to the User object, I decided to create a separate Login object as well which will handle the accepting of username/password and loading the character and creating a new User object from the loaded information. Additionally the Login object also handles creating a new account, posing the various questions (username, password, password confirmation etc), then creating a new User object based off of the data provided. This way a full User object isn't created until credentials have been verified, and also prevents User from showing up in the user list, or receiving broadcast messages etc before they have completed the Login process.

As part of the work to separate the Login class from the User class, I also created a separate 'Connection' class as a wrapper over top of the raw sockets. Primarily this just creates a couple of convenience functions over the Socket for write, writeLine, readLine, set the lineHandler and close the Socket. I made this its own library, so it is imported into the Server which creates the initial socket and then wraps the Connection around it, and then hands it off to the mudlib (which is its own library). The mudlib also imports the Connection class. The mudlib then passes the Connection off to a new Login object which uses that to communicate to the user (again as opposed to the raw socket). The connection object is then passed off to the newly created User object once it is created. User object adds a couple more convenience functions as well but mostly works with the Connection object as well.

I've added a couple more commands as well. Now able to 'say' to the local room, change the default prompt (which does save to the User file), and of course the always abused 'emote' command. Additionally I've got a good start on a simple line editor (and a simple command wrapper for it to see the results). The Editor is a separate object which will take over the Input/Output streams of a socket from the user. It does not remove the user itself, nor does it take the user's connection object. Rather it just updates the Connection object to use the Editor's lineHandler.

The editor has two modes, Command mode and input put. By default you start in command mode, from which you can either insert or append which will put you into the input mode. To exit the input mode, one just needs type a period (.) on a blank line. The editor works similarly to 'ed' but a little prettier. (For instance a prompt of ': ' in command mode and prompt of '~ ' in input mode). Currently implemented commands in the editor: a - append, i - insert, q - quit, h - help, p - print line, d - delete line. In the future some commands such as p and d will accept a range argument as well however this is yet not implemented. There is no 'save', if you quit it will return whatever is in the buffer.

The editor is called by the User object. The User object in turn is called by some other command or object (most likely a command which will do something with the output). To start the edit the User object has startEdit method called on it with a callback function as an argument. Once the editor finishes it calls the method doneEdit in the User object which resets the proper line handlers, and then calls the previous callback function with the String returned from the Editor.

When I first started working with the Editor I thought the obvious solution to use it would be a StringBuffer. However after only a few minutes of implementing it with the StringBuffer I found it lacked much of the basic functionality I needed, such as getting a specific line from the buffer, or removing a line from the buffer. I also can only add to the end of the Buffer and not to any particular point in the buffer. So that didn't last long at all.
When I looked at the StringBufferImpl class in the library, I saw that it was nothing more than a List of Strings. So I decided to do the same. So I created two List of strings. One for the full Buffer, and one for a temporary buffer. The temporary buffer is used in input/edit mode. Upon leaving input mode back to command mode then the temporary buffer is inserted into the full buffer.
To accomplish this, I originally tried using List#setRange. However I quickly discovered that wouldn't work as it would just overwrite the range, as opposed to inserting at that point. Additionally if the temporary buffer would exceed the length of full buffer, then it would throw an exception. Even though the list is an extendable list. So I tried List#insertRange, and that expanded the List as required, but it will only initialize the new range to the same value. You can't insert the range with with list as the new values. So it required first using insertRange to add the space for the temporary list, then setRange to change those values to be the strings in the temporary buffer.

Another area of interest is that I have implemented a RoomManager. Eventually the RoomManager will be expanded to have time outs so once a room has gone 'stale' (no activity in it for x number of minutes) it will reset the room. And, if still not used after y number of minutes, the object will be unreferenced for the GC to clean up as it will. The first part of this however, was trying to to determine how I can efficiently 'dynamically' load the rooms when someone tries to access them, or alternatively reference an already loaded room. Since I can't use eval to load a file at this point, the code has to be loaded on start up.
So I created two maps, each map has a key of a room identifier. One map holds the rooms themselves if they are loaded. The other, roomFuncs holds functions which create the room. Rooms are added by calling the add method on the RoomManager, which takes a callback as an argument. The add method then runs the call back to create the room. It gets the room's unique identifier to use as the key. Then it stores the callback in the roomFuncs map. Then when it comes time to access the room, we call the method putIfAbsent method on the room map, passing our roomId and a reference to roomFuncs[roomId] which returns the callback to create the room if the object is not already loaded. The putIfAbsent is a nice little method, and quite handy for this.

Well that's about all for now anyways. I must say I'm having a blast writing this simple mud, and while I have a long way to go for a full featured mud, it is virtually usable at this point. Dart is a lot of fun!

Monday, February 27, 2012

Dart MUD... Progress 1

So before I get started, I just want to put out there that the latest integration build of DartEditor has been released, and you should pick it up here. This build contains the latest IDE, the SDK and now Dartium as well! For those of you running 64-bit systems, it is a 32-bit build of Chromium so you may encounter some library errors if you have not installed the extra components you need.

So now that that news is out of the way.. a little progress report on my Dart MUD. (I may need to come up with a better development name, as it conflicts with the already existing MUD: Dartmud.com) So far I have a minimal server, from which I've managed to remove most unnecessary code from and make the mudlib independent of the server. I have a generic MudLib object which handles most interactions between server and specific mudlib components. A User object which handles the client connections and various user properties (username, passwords, etc). However I'm not completely happy having the socket implementation directly part of the User object, and may break down into a connection object which the User class will either have it as a property, or may inherit from (just because someone connects, doesn't mean it will result in a User on the system).

After some playing around with a static class which contained the various commands a player can execute, I decided to instead create a small class, and each command is an instance of the class. All of which are managed by a CommandDaemon/Command Manager. I can see this allowing for more flexibility in adding commands globally and perhaps even in just specific rooms or objects. On the note of Rooms and GameObjects, I have classes for these also implemented, as well as a 'container' interface and a factory class for it. Users and Rooms both currently extend the factory class for the container.

At this point, I've added a few 'commands', including the exit and shutdown commands from the EchoServer (though now setup to be used with the Command Manager I have implemented). I've also added a basic 'look', 'broadcast' and 'help' command, and am in the process of fleshing out these respective commands so they properly handle any arguments sent to them (broadcast works to notify all users, regardless of if they are in the same room as the person issuing the commands.

I still have so much to do. Currently when connecting you are prompted for a username/password to log in. However you can enter anything an be granted access with the username you enter. I need to be able to create a new user, to save the user, and to load them when logging in. Also I have a very basic room, but only the one. I still have to create more rooms and implement travel commands to move between rooms. To say nothing of the other normal things you would expect on a mud including equipment, objects, money, combat, etc etc etc.

While I still have a very long way to go to make a basic mud, I'm quite pleased with the progress I have been making towards that end. Something I may want to look towards as well is some type of 'reload' or 'refresh' command, as I have to shutdown the server and restart it after each change I make. I'll also need to work on a memory management daemon of some kind too, so that once the mud gets to a decent size, objects will only be loaded/created when accessed, and after a period of time, removed if not used or near an active player.

One of these days I'm going to make an effort to learn git/git hub too.

Thursday, February 23, 2012

Featureless Sockets

Okay, so the title is a bit of a stretch. Dart Sockets are far from featureless. As some of you may or may not know, I've decided to take a segue in my EchoServer towards working on a basic MUD and MUDLib written in Dart. It will be far from full featured, and far from fully functional but it will be the basis and a nice little start.

Within a few minutes of converting the EchoServer to be the basis of the MUD driver, I've already run into an issue of a missing feature with Sockets in Dart. I'm unable to get the host of an incoming connection. That is, I can't find out who is connecting to me. While it's a trivial matter for a MUD, or many other services, one would expect such functionality for various uses including HTTP server logging, etc.

After digging through the bleeding edge to see if it may have been recently included but just not yet updated in the API, I still can't find any reference to such properties. So as such, I've created a bug report over in the Dart issue tracker. If you're interested, or want to star the report you can view it here at: Issue 1819

As I progress with the Dart core and MUDLib I'm sure I'll find some additional bugs to report, or just missing features I can think could be added. Additionally I'll keep progress reports available on here and eventually release the source on GitHub or something similar.

Tuesday, February 21, 2012

Serving Sockets... The Salad

I haven't had the opportunity to do much with Dart lately, due to scheduling. However this afternoon I had a little chance to write some more. I decided that I wanted to modify the EchoServer to maintain the connection and close the connection only when receiving a specific command. I also wanted to add a command to shutdown the server itself completely. In order to properly accommodate this, I needed to refactor the server code a little. I wrapped the ServerSocket in a manager class:

#import('dart:io');

class ServerManager {
  ServerSocket _listenServer;
  
  ServerManager() {
    _listenServer = new ServerSocket("127.0.0.1", 5700, 0);
    
    _listenServer.connectionHandler = this._handleConn;
  }

  void _handleConn(Socket conn) {
    StringInputStream clientIn = new StringInputStream(conn.inputStream);
    
    clientIn.lineHandler = () {
      String input = clientIn.readLine();
      print("Received: $input");
      if(input.toLowerCase() == 'stop') {
        String cls = "** Stopping Server. Closing connection **\n";
        conn.writeList(cls.charCodes(), 0, cls.length);
        conn.close();
        _listenServer.close();
        print("** Stopping Server! **");
      } else if(input.toLowerCase() == 'exit') {
        String cls = "** Closing connection to client **\n";
        conn.writeList(cls.charCodes(), 0, cls.length);
        conn.close();
        print("** Closing connection to client. **");
      } else {
        String output = "${input.toUpperCase()}\n";
        conn.writeList(output.charCodes(), 0, output.length);
        print("Sent: $output");
      }
    };
    
  }
}

void main() {
  ServerManager sMan = new ServerManager();
 
}

So as you can see I made a few changes from my original EchoServer. As mentioned above I wrapped the server in a manager class, this enables me to easily close the server socket without using a global variable. In addition I added a couple clauses to check for the 'stop' or 'exit' commands which will stop the server or just close the client connection respectively. And finally I stopped pulling the output stream of the sockets directly, and instead use the writeList methods directly on the socket itself. I wasn't gaining any real benefit by creating an additional variable for the socket's OutputStream, so I just dropped it altogether.

Now as is, the above will run and accept connections and echo any new lines until the stop or exit commands are received. If the exit command is received, then the server will close the connection to that socket. If stop is received it will close the connection to that socket and then tell the server to close. However because of the event driven nature of the server, the Sockets and ServerSockets are not blocking. That is, even without adding any additional threads (Isolates), we can accept connections from multiple sources. If you open up multiple telnet connections to the host, you can see how you can send data and receive responses on each connection independent of the other.

But this also leaves us with a small issue. If we tell the server to stop from one telnet session while the other is still active.. the server will accept the stop command, and it will schedule the ServerSocket to be stopped, but not until the other socket has been closed. Try it out and you will see that the connection in which we issue the stop command is disconnected, and the console will indicate that the server is stopping. But the other telnet session will remain active until we issue an exit or stop command. Only once the 2nd session is closed will the server stop. And if for some reason the other session does not terminate properly (for instance connection drops or the telnet application is closed before issuing an exit/stop command) then the server will hang, not accepting new connections but not terminating either (assuming eventually the socket will time out but potentially not since I do not have those error handlers in place either).

This may be the desired situation with some servers to shut them down gracefully for instance, however in our EchoServer we want it to shut down immediately if it receives the stop command. So we'll need to keep a list of active connections and iterate through them and close each one, then stop the server. So I ended up with the following:

#import('dart:io');

class ServerManager {
  ServerSocket _listenServer;
  List _socketList;
  
  ServerManager() {
    _socketList = new List();
    _listenServer = new ServerSocket("127.0.0.1", 5700, 0);
    
    _listenServer.connectionHandler = this._handleConn;
  }
  
  void sendStops() {
    List cls = "** Server received stop request. Closing connection to client **\n".charCodes();
    
    while(!_socketList.isEmpty()) {
      Socket conn = _socketList.removeLast();
      conn.writeList(cls, 0, cls.length);
      conn.close();
    }
  }
  
  void _handleConn(Socket conn) {
    _socketList.add(conn);
    StringInputStream clientIn = new StringInputStream(conn.inputStream);
    
    clientIn.lineHandler = () {
      String input = clientIn.readLine();
      print("Received: $input");
      
      if(input.toLowerCase() == 'stop') {
        sendStops();
        _listenServer.close();
        print("** Stopping Server! **");
      } else if(input.toLowerCase() == 'exit') {
        String cls = "** Closing connection to client **\n";
        conn.writeList(cls.charCodes(), 0, cls.length);
        int sockInd = _socketList.indexOf(conn);
        _socketList.removeRange(sockInd, 1);
        conn.close();
        print("** Closing connection to client: $sockInd **");
      } else {
        String output = "${input.toUpperCase()}\n";
        conn.writeList(output.charCodes(), 0, output.length);
        print("Sent: $output");
      }
    };
    
  }
}

void main() {
  ServerManager sMan = new ServerManager();
 
}

As you can see I also added a method sendStops just to iterate through all the sockets, popping them out of list and sending them the stop notice and disconnecting them. I made this separate from the actual stopping of the server in case it should ever be required for any other reason as well. Initially I tried using a Set to hold just unique connections, and provide easier way of removing elements however I found out that there's an issue with Set's in that any values stored in a set must implement Hashable. This wasn't added in the API documentation and it was only after a little digging through the DartBug page and Newsgroup that I found this is 'expected' behaviour. As such, I had to use the list. For a specific 'exit' command I have to get the index of the value and remove it from the list with removeRange with a size of 1 element. I also setup the broadcast message directly to a List of Int's immediately just to avoid having to convert it multiple times as I iterate through the connections. While still missing any error handling, etc. I'm rather pleased with how the server is progressing and in some ways it conjures up images of the old school MUD's. Maybe a project to play with?

Thursday, February 16, 2012

Serving Sockets... The Soup Crackers

So as I mentioned in my previous post, there is more than one way to communicate through sockets. Below I have show 3 different, though similar ways to make use of sockets. This is far from exhaustive and please note that this is for demonstration purposes only and as is, should not be used in production code of any kind. They're missing error checking, stream verification, etc.

I am interested if any one has any comments or feedback as to use cases where they would choose to use one of the particular methods over another. Everything below is fairly procedural and could probably be implemented much cleaner as Objects, but as mentioned this is purely a test case using the EchoServer I wrote a couple posts back.

#import("dart:io");

void main() {
  sockets_with_handlers();
  
  sockets_with_direct_streams();

  sockets_with_socket_streams();
}

void sockets_with_handlers() {
  Socket usingHandlers = new Socket("127.0.0.1", 5700);
  String test = "SocketHandlers Test String\n";
  
  // Called when we successfully connect
  usingHandlers.connectHandler = () {
    print("Handers: Connected");
  };
  
  // Called when we can write to the socket.
  usingHandlers.writeHandler = () {
    usingHandlers.writeList(test.charCodes(), 0, test.length);
    print("Handers Sent: $test");
  };
  
  // Called when we receive from the socket.
  usingHandlers.dataHandler = () {
    int availBytes = usingHandlers.available();
    List buff = new List(availBytes);
    usingHandlers.readList(buff, 0, availBytes);
    print("Handers Received: ${new String.fromCharCodes(buff)}");
  };
  
  // Called when the input stream from socket is closed.
  usingHandlers.closeHandler = () {
    print("Handers: End of stream. Closing connection");
    usingHandlers.close();
  };
}

void sockets_with_direct_streams() {
  Socket usingStreams = new Socket("127.0.0.1", 5700);
  String test = "Direct Socket Streams test string\n";
  
  // To be 'pure' without the handlers we assume connection is successful.
  // NOTE: Not recommended!
  print("Direct Streams: Connected");
  // Wrap the input stream in StringInputStream so we can
  // make use of the convenience functions.
  StringInputStream inStream = new StringInputStream(usingStreams.inputStream);
  OutputStream outStream = usingStreams.outputStream;
  
  // Write to our stream
  outStream.write(test.charCodes());
  print("Direct Streams Sent: $test");
  
  // Use our handle wrapper to read lines
  // Saves us from using dataHandler directly
  inStream.lineHandler = () {
    String input = inStream.readLine();
    print("Direct Streams Received: $input");
  };
  
  // All bytes have been read and input stream is closed.
  inStream.closeHandler = () {
    print("Direct Streams: End of stream. Closing connection");
    usingStreams.close();
  };
}

void sockets_with_socket_streams() {
  Socket usingSocketStreams = new Socket("127.0.0.1", 5700);
  String test = "Socket In/Out Streams test string\n";
  
  // To be 'pure' without the handlers we assume connection is successful.
  // NOTE: Not recommended!
  print("Socket Streams: Connected");
  
  SocketInputStream inStream = new SocketInputStream(usingSocketStreams);
  SocketOutputStream outStream = new SocketOutputStream(usingSocketStreams);
  
  // Write to our stream.
  outStream.write(test.charCodes());
  print("Socket Streams Sent: $test");
  
  // Called when new data arrives in our SocketInputStream
  inStream.dataHandler = () {
    List buff = inStream.read();
    print("Socket Streams received: ${new String.fromCharCodes(buff)}");
  };
  
  // Input stream has been closed. Make sure output stream
  // and socket itself are also closed.
  inStream.closeHandler = () {
    print("Socket Streams: End of stream. Closing connection");
    outStream.close();
    usingSocketStreams.close();
  };
}
Direct Streams: Connected
Direct Streams Sent: Direct Socket Streams test string

Socket Streams: Connected
Socket Streams Sent: Socket In/Out Streams test string

Handers: Connected
Handers Sent: SocketHandlers Test String

Direct Streams Received: DIRECT SOCKET STREAMS TEST STRING
Socket Streams received: SOCKET IN/OUT STREAMS TEST STRING

Handers Received: SOCKETHANDLERS TEST STRING

Direct Streams: End of stream. Closing connection
Socket Streams: End of stream. Closing connection
Handers: End of stream. Closing connection

One of the first things you'll notice with the output is that due to the non-blocking nature of the callbacks, some of the calls happened in near-parallel. In the sockets_with_socket_streams function, while I didn't specifically make use of it, we could have also wrapped the SocketInputStream with a StringInputStream as well, similar to what we did in the sockets_with_direct_streams function.

Serving Sockets... The Soup

Alright, first thing first. Announced during yesterday's Episode 2 of Dartisans Hangout, Dartium binaries are being relased. Initially for Mac OS X and Linux with Windows binaries to follow soon. I won't go on about this much as it has already been mentioned again and again, and again.

One thing to note however, is that currently the Linux Build is a 32-bit version so if you're running a 64-bit OS you will need to ensure you have the proper 32-bit libraries downloaded and installed. Or alternatively, wait for the 64-Bit builds to arrive.

Continuing with my small series on Sockets in the Dart:IO library, I thought well I have an EchoServer written, albeit extremely primitive, my next 'logical' step is to make a client which connects to the server, sends a string, receives the echo response and disconnects. This should be pretty straight forward, since it was so easy to make the server... Oh boy was I ever wrong about that. Not that Sockets are overly complex or anything, they just have many more ways of accomplishing the same thing, but they can't be used in conjunction as I found out with the following:

#import("dart:io");

void main() {
  // Create a new socket connecting to localhost and port 5700
  // the same port as our echo server we wrote is running on.
  Socket conn = new Socket("127.0.0.1", 5700);
  StringInputStream inputStr;
  OutputStream outputStream;
  
  // method is called when connection is established.
  conn.connectHandler = () {
    print("Now Connected");
    inputStr = new StringInputStream(conn.inputStream);
    outputStream = conn.outputStream;
    
    String test = "This is a simple test\n";
    outputStream.write(test.charCodes());
    print("Sending: $test");
    
    // We wrapped the input stream for this easier reading
    // using the lineHandler as opposed to bulk data.
    inputStr.lineHandler = () {
      String input = inputStr.readLine();
      print("Recieved: $input\n");
    };
  };
  
  // Called when the last byte of data has been read from the socket.
  // Socket potentially still open for writing.
  conn.closeHandler = () {
    print("Connection closed. Last byte of data has been read from stream");
    conn.close();
  };
  
}
Now Connected
Unhandled exception:
StreamException: Cannot get input stream when socket handlers are used
 0. Function: '_Socket@14117cc4.get:inputStream' url: 'dart:io' line:4231 col:9
 1. Function: '::function' url: 'src/dart/echo/EchoClient.dart' line:13 col:54
 2. Function: '_Socket@14117cc4.firstWriteHandler' url: 'dart:io' line:4274 col:64
 3. Function: '_SocketBase@14117cc4._multiplex@14117cc4' url: 'dart:io' line:3898 col:23
 4. Function: '_SocketBase@14117cc4.function' url: 'dart:io' line:3996 col:59
 5. Function: 'ReceivePortImpl._handleMessage@924b4b8' url: 'bootstrap_impl' line:1734 col:22

So as we can see, this didn't work quite as cleanly as we'd have liked. Apparently the issue above is with my mixing the Socket handlers, such as connectHandler and closeHandler, while also pulling the socket's input and output streams themselves. In the case of the InputStream wrapping it in a StringInputStream handler to make dealing with input a little easier since we only expect text.

Initially I thought this error was telling me that I cannot use any socket handlers, if I wanted to pull the IO streams directly from the Socket. However I only just now stumbled upon an easier solution. The error isn't with the use of connectHandler and closeHandler on the socket. Apparently only the closeHandler is triggering the error. So I can move that method to be called on the StringInputStream instead. So a quick re-write gives me the following:

#import("dart:io");

void main() {
  // Create a new socket connecting to localhost and port 5700
  // the same port as our echo server we wrote is running on.
  Socket conn = new Socket("127.0.0.1", 5700);
  StringInputStream inputStr;
  OutputStream outputStream;
  
  // method is called when connection is established.
  conn.connectHandler = () {
    print("Now Connected");
    inputStr = new StringInputStream(conn.inputStream);
    outputStream = conn.outputStream;
    
    String test = "This is a simple test\n";
    outputStream.write(test.charCodes());
    print("Sending: $test");
    
    // We wrapped the input stream for this easier reading
    // using the lineHandler as opposed to bulk data.
    inputStr.lineHandler = () {
      String input = inputStr.readLine();
      print("Recieved: $input\n");
    };

    // Called when the last byte of data has been read from the socket.
    // Socket potentially still open for writing.
    inputStr.closeHandler = () {
      print("Connection closed. Last byte of data has been read from stream");
      conn.close();
    };
  };
  
}
Now Connected
Sending: This is a simple test

Recieved: THIS IS A SIMPLE TEST

Connection closed. Last byte of data has been read from stream

Well then, that's working so much better. And in fact I could leave it here.. and will for the time being. However later, perhaps tomorrow, I will write a continuation of this post on some of the other methods of using and communicating with sockets. They are plentiful.

Wednesday, February 15, 2012

Servering Sockets... The Appetizer

So I decided it was time to start playing around with some Sockets. As we already know, I'm not a huge fan of the chat sample that's available, as I find it a little too much. Fortunately the Sockets and ServerSockets are not too different from the Process API, so keeping that in mind I wrote a dead simple echo server. This does not adhere to any protocols or anything of that nature. It just accepts input from a client and sends it back to the client. Truth be told, I should be a little ashamed of the following code as its far from complete, far from safe, and far from extendable. But it was a good starting point and I'll be able to now take it and re-factor it convert it into something usable. The goal was simple, to write a simple server and have evidence that communication was bi-directional.

So first I'll provide the code and then a little discussion about it.

#import('dart:io');

void main() {
  // Create a server and bind it to an address and port number.
  ServerSocket listenServer = new ServerSocket("127.0.0.1", 5700, 0);
  
  // This is called when a connection is received.
  listenServer.connectionHandler = handleConn;
}

// Socket conn is the incoming connection.
void handleConn(Socket conn) {
  // Wrap the input from the connection around a StringInputStream to make reading it easier.
  StringInputStream clientIn = new StringInputStream(conn.inputStream);
  // Get the output stream too so we can write back to the client.
  OutputStream clientOut = conn.outputStream;
  
  // Line handler is called when a new line is received to the StringInputStream
  // which we've wrapped around the InputStream from the socket.
  clientIn.lineHandler = () {
    String input = clientIn.readLine();
    print("Received: $input");
    String output = "${input.toUpperCase()}\n"; 
    clientOut.write(output.charCodes());
    print("Sent: $output");
    conn.close();
  };
  
}

So as you can see above the code is very short and simple. I recommend running this in the IDE as currently the process will run forever. Later on I may add a command to shutdown the server from the client. To test out the above code start the server with the DartVM then open a telnet session. Telnet to 'localhost:5700' Then simply type a line of text and hit enter. You should get the same line of text sent back to you but uppercase.

Basically we create the server, and then just handle any incoming connections with it. The connectionHandler is called when a new connection is established to the server, and the function is passed the Socket to the incoming connection.
From that socket we get the InputStream and OutputStreams. For the InputStream we wrap it in a StringInputStream which simplifies reading strings from the connection, and since we're not really worried about receiving any bulk or binary data as such.

Next, thanks to the StringInputStream, we setup a lineHandler which is called when a new line of text is received by the stream. Then we use readLine to store that line in a variable.

We print what we've received to the console, then create a new string which is uppercase version of the line we just received, and add a new line character to the end of it. We want to add the newLine character because when we use the readLine method, it strips the new line character for us automatically. In this case I wanted to return an uppercase string of the original just for visual confirmation that what is being returned is different in some way from the original so it can't be blamed as a local echo by the telnet client or anything like that.

Next we write the output back to the output stream. Because the OutputStream is not wrapped by a String stream we need to convert our string into a list of character codes which are then in turn sent back through the SocketOutputStream. We also write to the console what we sent.

Finally we close the socket. In a later example we may leave this open until we send a specific comment to terminate the client connection, or specifically have the client close the connection instead.

As I mentioned, this example is pretty much as simple as you can get. It also is missing any error handling, any protocol conformation, but it gave me a comfortable start/introduction to sockets in general. If you have any suggestions or comments regarding the code by all means I eagerly look for feedback or suggestions.

Wednesday, February 8, 2012

String that Process

So a comment from Florian Loitsch on my previous post indicated that as opposed to creating a new string from character codes as I was doing for handling output of the process, I can use the existing StringInputStream to automatically wrap the basic input stream into a string. As indicated by the API I can then pull the strings on a per-line basis or as a string chunk. I've revised my example to put both methods to use. Before commenting on it, lets see the code:
#import("dart:io");

void main() {
  // Create a new process with list of arguments and start it.
  Process proc = new Process.start('/usr/bin/snmpget', ['-v', '2c', '-c', 'public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  
  // Once started, handle the process including receiving input.
  proc.startHandler = beginHandle() {
    print("Inside of start Handler");
    
    // Create StringInputStreams from the basic input streams of the process.
    StringInputStream res = new StringInputStream(proc.stdout);
    StringInputStream err = new StringInputStream(proc.stderr);
    
    // A handle a new line of text added to the input stream (stdout).
    // readLine does not include the terminating new-line character
    res.lineHandler = showLine() {
      print("Inside InputStringStream lineHandler");
      print(res.readLine());
    };
    
    // Handle new data sent to the input stream (stderr)
    // This happens whenever any data is available and does not
    // wait for a new line.
    err.dataHandler = errRead() {
      print("Inside InputStringStream dataHandler");
      print(err.read());
    };
  }; // End off Proc.startHandler
  
  // When the process has ended this handler is called
  // passes process's exit code.
  proc.exitHandler = exitHandle(int exitCode) {
    print("Inside of exit Handler");
    print("Exit code is: ${exitCode}");
    proc.close();
  };
  
  // Called when an error occurs when spawning the process.
  proc.errorHandler = errorHandle(ProcessException errorMsg) {
    print("Inside of Error Handler");
    print(errorMsg);
  };
}

I've also added some comments to the above to help clear it up a little as well. With the particular process I'm launching generally I'm only interested in parsing one return line at a time. So as such with my StringInputStream for stdout I decided to use the lineHandler method. Because stderr may produce more than one line of error, I used the dataHandler trigger the reads. Note that the two are mutually exclusive. That is the same StringInputStream can only define a lineHandler or a dataHandler. Not both.

The lineHandler is called each time a new line is available to the stream. dataHandler is called anytime there's new data available, no matter if it contains a new line or not. This is good if you want to receive a bulk of a string, or potentially expect an unterminated string.

Note I also did not methods for the errorHandlers in this simple example, nor verify if the stream is currently open. Using both this method and the previous one of directly using the basic InputStreams are roughly the same size, however unless you explicitly require the raw character codes, I would personally suggest using the StringInputStream over the basic InputStream for potential performance gains and to help future-proof your code against any internal library chances which make affect results or performance down the road with the InputStream and string from character codes.

Tuesday, February 7, 2012

The INs and OUTs of IO.

So, as per my previous post, I've started to play with the dart:io library. While there is a sample of its usage to create a chat server available in the bleeding edge repository, but in all honesty that is far more complete and thorough than anything I plan on just yet. So essentially I'm just going off of the API documentation. Which, while good, is not exactly bursting with usage examples.

As I mentioned in my previous post, currently the goal of the script is to just run an external command and output the results. The command I chose to work with in this case is snmpget for conducting SNMP queries on equipment. This is a precursor to some further scripting I would do with the VM and Dart in general on both administrative fronts and potentially a full web-app as well.

So I focus my attention on the Process API. Now before I go any further I want to call to attention that I am far from an expert in Dart, or even Javascript for that matter. Most of my programming experiences lay in other area such as perl, python, and then Ruby. The particular use of callbacks in the manner they are used is something not entirely familiar with me and I am quite possibly breaking a few rules with them. If you have any notes or suggestions on improving/securing my simple code by all means feel free to leave some feedback in the comments!

So looking at the API I see I obviously need to create a new Process object with Process.start. I will then also need to have a startHandler and we'll throw in a simple errorHandler just incase I mess up something. Looking at the information for close I should also put in an exitHandler. In fact the close information gives me my first hint as to where the process logic should be contained. According to the documents for close, Usually, close should be called in the exitHandler. That tells me most of my other logic dealing with input/output streams should probably be inside of the startHandler. I also see I'll need to get the stdout of the process to print it. So I start with this simple code and see where I can get from there (note for security reasons I've changed the community code, and IP of the device).
#library('snmp_poll');

#import("dart:io");

void main() {
  var proc = new Process.start('/usr/bin/snmpget', ['-v 2c', '-c public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  proc.startHandler = beginHandle() {
    var res = proc.stdout;
    print(res.read());
  };

  proc.errorHandler = errorHandle(errMsg) {
    print(errMsg);
  };

  proc.exitHandler = exitHandle(exitCode) {
    print(exitCode);
    proc.close();
  };
}
null
1

Initially I had some issues with assigning the handlers but that's because I tried to use the startHandler, errorHandler, and exitHandler as methods as opposed to properties with get/set. That was simple enough to straighten up so didn't bother showing that mistake here.
So after reading the API for InputStream, I see I should retrieve the data with the read method. But all I'm getting is null. Also, my return code is 1 which tells me it didn't execute properly. The process itself did execute which is why it's not calling the errorHandler. But still there's something missing. No output and why am I getting a 0 for the return code. Time to add some more code and a few debugging prints.
#import("dart:io");

void main() {
  var proc = new Process.start('/usr/bin/snmpget', ['-v 2c', '-c public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  proc.startHandler = beginHandle() {
    var res = proc.stdout;
    var err = proc.stderr;
    
    print("Inside beginHandle");
    print(res.read());
    print(err.read());
  };

  proc.errorHandler = errorHandle(errMsg) {
    print("Inside beginHandle");
    print(errMsg);
  };

  proc.exitHandler = exitHandle(exitCode) {
    print("Inside beginHandle");
    print("Exit code is: $exitCode");
    proc.close();
  };
}
Inside beginHandle
null
null
Inside beginHandle
Exit code is: 1

Okay so now both stderr and stdout are both showing null. Gah what's going on. I've done something wrong obviously. Reading the API for InputStream tells me that if no data is available null is returned. But why is there no data available. Standard out or Standard error should be reporting something here.. All I can think is that we need to use the dataHandlers because these are streams rather than dumped to a string as I'm used to with other scripting languages (particularly when using the back-tick notation). Alrighty then, lets add the dataHandlers.
#import("dart:io");

void main() {
  var proc = new Process.start('/usr/bin/snmpget', ['-v 2c', '-c public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  proc.startHandler = beginHandle() {
    print("Inside beginHandle");
    var res = proc.stdout;
    var err = proc.stderr;
    
    res.dataHandler = nowRead() {
      print("Inside stdout dataHandler");
      print(res.read());
    };
    
    err.dataHandler = errRead() {
      print("Inside stderr dataHandler");
      print(err.read());
    };
  };

  proc.errorHandler = errorHandle(errMsg) {
    print("Inside beginHandle");
    print(errMsg);
  };

  proc.exitHandler = exitHandle(exitCode) {
    print("Inside beginHandle");
    print("Exit code is: $exitCode");
    proc.close();
  };
}
Inside beginHandle
Inside stderr dataHandler
_InternalByteArray
Inside stderr dataHandler
_InternalByteArray
Inside stderr dataHandler
_InternalByteArray
Inside stderr dataHandler
_InternalByteArray
Inside stderr dataHandler
_InternalByteArray
Inside beginHandle
Exit code is: 1

Alright now we're getting somewhere... except apparently print doesn't like the output from the read method. Though looking at the API I do see that it does return a List. So lets iterate over that list to see what's in them. Since I see its only the stderr reporting the results I'll just play with that one now. And to save some screen space I'm just going to paste the snippet below:
...
err.dataHandler = errRead() {
  print("Inside stderr dataHandler");
  for(var x in err.read()) {
    print(x);
  }
};
...
Inside beginHandle
Inside stderr dataHandler
73
110
118
< ... snipped for brevity ... >
50
99
10
Inside beginHandle
Exit code is: 1

Ack! Okay that's so not what I wanted.. I'm getting integer character codes instead of the strings or characters returned. I need to convert these character codes to a new string. Fortunately a little search found just the constructor I needed in the String class. new String.fromCharCodes. So a quick little adjustment to my script and I can actually see what error I'm receiving from the snmpget command itself. And I no longer require that silly for loop.
#import("dart:io");

void main() {
  var proc = new Process.start('/usr/bin/snmpget', ['-v 2c', '-c public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  proc.startHandler = beginHandle() {
    print("Inside beginHandle");
    var res = proc.stdout;
    var err = proc.stderr;
    
    res.dataHandler = nowRead() {
      print("Inside stdout dataHandler");
      print(new String.fromCharCodes(res.read()));
    };
    
    err.dataHandler = errRead() {
      print("Inside stderr dataHandler");
      print(new String.fromCharCodes(err.read()));
    };
  };

  proc.errorHandler = errorHandle(errMsg) {
    print("Inside beginHandle");
    print(errMsg);
  };

  proc.exitHandler = exitHandle(exitCode) {
    print("Inside beginHandle");
    print("Exit code is: $exitCode");
    proc.close();
  };
}
Inside beginHandle
Inside stderr dataHandler
Invalid version specified after -v flag:  2c

Inside stderr dataHandler
USAGE: snmpget [OPTIONS] AGENT OID [OID]...
<... snipped for brevity ...>

Alright so I'm getting the full snmpget error message but at the top I can see it's not happy about my flags. After playing around a little I discovered that this has something to do with how exactly the Process.start passes the argument list. The two options I came up with were to remove the spaces (which can sometimes cause issues with the receiving program) or after each flag place its argument as its own member of the list. So making that change I end up with the following code which works perfectly and calls the snmpget command and prints its return.
#import("dart:io");

void main() {
  var proc = new Process.start('/usr/bin/snmpget', ['-v', '2c', '-c', 'public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  proc.startHandler = beginHandle() {
    print("Inside of start Handler");
    var res = proc.stdout;
    var err = proc.stderr;
    res.dataHandler = nowRead() {
      print("Inside stdout dataHandler");
      print(new String.fromCharCodes(res.read()));
    };
    err.dataHandler = errRead() {
      print("Inside stderr dataHandler");
      print(new String.fromCharCodes(err.read()));
    };
  };
  proc.exitHandler = exitHandle(exitCode) {
    print("Inside of exit Handler");
    print("Exit code is: ${exitCode}");
    proc.close();
  };
  
  proc.errorHandler = errorHandle(errorMsg) {
    print("Inside of Error Handler");
    print(errorMsg);
  };
}
Inside of start Handler
Inside stdout dataHandler
iso.3.6.1.2.1.1.1.0 = STRING: "Super Top Secret Equipment Name"

Inside of exit Handler
Exit code is: 0

Finally! Success! This is just the start of the script which I can now use to handle, parse and interpret the results of the snmp queries I make and pass them along as needed, perhaps providing them via json as an ajax service. In any case I certainly learned a lot more than expected about how Dart handles I/O and how I, in turn, need to handle the requests. While this small sample in itself seems rather verbose I can see how a few small wrappers could make dealing with a large number of processes fairly clean and robust.

Library hide and seek

So today I took some time to start playing with the IO library in Dart. Right off the bat I started running into problems. First off, I'm running Linux 64-bit and I'm using the latest integrated version of the IDE and SDK (Build 3934). Taking a look into the SDK I do see the IO libraries and such so I couldn't wait to get started. My first little script just spawns a new process running and displays the results. In this case in particular I'm spawning an snmpget command to run a quick and simple snmp query.

So I fire off my quick and dirty little script to see if I can get any output. I save my file and... err what what? The editor is giving me a nasty little error. Right there on line one of all places (my library declaration), a few other errors follow as well.
File not found: dart:io

A little further down is:
Multiple markers at this line
  - no matching constructor for Object
  - cannot find type Process

This line is where I make my new Process object using new Process.start(). Based on the first error, I gather that this line is generating an error because the IDE isn't recognizing any class called Process for me to create a new object from so it's trying to use the base Object class, which doesn't contain a named constructor called 'start'.

I mess around in the IDE to verify that the SDK libraries are available and that it is seeing the SDK as installed. Yeah, everything seems hunky-dory there. And... wait... my library panel is displaying the dart:io library right there in the left-hand pane. Something here doesn't seem right at all. I do a quick search in the dartbugs but don't find anything. So I came up with the following quick trial case and submitted a bug report.
#import("dart:io");

void main() {
  print("Hello World");
}

Save the above in the IDE and you should see the error generated at line 1. As I mentioned, I filed a bug report: Issue 1577. At first I thought that would be the end of my trials today, however I hit the run button anyways and low and behold the console is showing expected output. So while the IDE isn't recognizing the library in the editing pane, it is recognizing it in the libraries (though showing errors in it none-the-less) and the dart VM does run the code as expected. Excellent maybe I will get to do some testing after all!

Continued in INs and OUTs of IO.

Monday, February 6, 2012

New Editor and SDK

Just today I was thinking "There's been a large number of changes to the SDK lately and the integration builds of the SDK and Editor haven't been updated. Maybe I'm going to have to start trying the continuous builds..." when low and behold, a thread on the Dartlang misc group indicates that there is a new integration build available. Build number: 3934. So off I go and download that and the new SDK build.

Beautiful. First a couple of things to note: I extracted the editor into the dart/ directory it was in, into my home directory. I extracted the SDK into my home directory as well and left it in the dart-sdk directory. I launched the Editor and in the launch preferences pointed the VM to the /dart-sdk/bin/dart executable. Which worked fine, but the SDK and compiler preferences said they didn't see the SDK installed and thus indicated that frogc would not work correctly. Then I recealled that I had seen somewhere previously in the mailing group that one should extract the dart-sdk into the dart editor directory. So I moved the dart-sdk directory into dart/, relaunched the editor and low and behold under the launch preferences it says there are no launch settings available... that's interesting. I then go in and check the SDK and Compiler preferences. It shows that I have SDK build number 3934 installed. Yay. It's seeing it. But odd there's no launch preferences.....

Or so I thought. Wow. Look at all those options... I opened up a small test file I had from the past and click the little drop down arrow next to the Run button... maybe they added the checked mode flag I had mentioned before. Whoa.. what's all this? Manage launches and I have a nice little dialog to "create an manage launches". Hitting the new button I see I can create a Dart launch in Dartium (don't have that installed at the moment but that's alright, Dart Server launch (primarily what I use for dealing with the dart VM directly) and Dart Web Launch. I go ahead and create a new server launch. Give the launch a name 'Run Test' in this case. Hit browse from the script and choose the only script in this project anyways. But its interesting that it lets you choose which file is the one that contains the main method in such an easy manner. And it appears (though I've yet to test it) that it will only list files associated with your current project). Any arguments I want to pass to the script as well and finally to run in checked mode or not. Wow very nice. I go ahead and create a separate launch just for running in checked mode as well just because I can. I must say I really like that launch interface in this new editor.

And now the SDK also has the IO libraries available to me as well so I will soon have to start playing with these to work on some of the scripting aspects I'm interested in using Dart for, server side.

Thursday, February 2, 2012

Few Updates

So I haven't had the opportunity to write much lately. There's been a lot going on a work and home that have made my time far too sparse. Hopefully I can start getting some of that stuff settled soon. In the mean time there's been lots going on with Dart for me to get caught up with.

Firstly, we're starting to see some server side related API's coming out. Including file and directory libraries and the start of some networking and socket libraries. They are only available in the continuous builds at the moment, however I expect to see them in the integration builds soon (speculation not based on any actual evidence). These are something I was most looking forward to, for both server side scripting and even just generic scripting possibilities. I think in the near future I will try to convert some of my existing applications and scripts from their various options (Javascript, PHP, Ruby) to Dart to see what issues I run into.

With the addition of the new libraries, we see a mass of updates to the current API documentation. Thanks to Mozilla's work documenting the DOM (And releasing it as CC) much of this information is now available in the Dart APIs. Check them out and become enlightened!

Speaking of becoming enlightened, if you have played with Javascript in the past, or perhaps are a bit of a Javascript Guru, check out the Dart Synonym page. Lots of little snippets that show how you accomplish something in Javascript and how you would do the same in Dart. It even includes a small subsection on comparisons between JQuery and Dart.

Also, be sure to check out the new Dart news and Updates site.

Monday, January 23, 2012

Wat?

Yesterday, Seth Ladd posted a link to a humorous video which highlights a few WAT?!?! moments in Javascript and Ruby. He then discussed how Dart deals with some of these moments in Javascript WAT moments and Dart. Please watch the video and read Seth's blog prior to reading.

There are WAT moments in virtually every language. A line from the video says something along the lines of So since it is the plus operator, we can expect the same result if we switch the operands around.. In the case of the video it was discussing how Javascript handles adding an array and a hash, and a hash and an array. However this reminded me of the issue I ran into myself already in experimenting with Dart. Look at the following code:
main() {
  print("wat" + 1);
  print(1 + "wat");
}

What is particularly interesting is running this example in the variety of locations. First lets run it from the Dartboard, without checked mode. We get the result:
wat1
1wat

More or less what you'd expect to get adding a string and a number. We do see a little warning off to the side indicating String is not assignable to num. But being not in checked mode it does run and because its Dartboard it falls back to the Javascript implementation of number + string. Well lets see what we'll get in the DartVM...
wat1
Unhandled exception:
NoSuchMethodException - receiver: 'wat' function name: 'addFromInteger' arguments: [1]]
 0. Function: 'Object.noSuchMethod' url: 'bootstrap' line:360 col:3
 1. Function: 'IntegerImplementation.+' url: 'bootstrap_impl' line:1469 col:32
 2. Function: '::main' url: '/home/mbutler/dart/Test/Test.dart' line:3 col:11

Err WAT? Even more, that runtime error is.. far from clear on just what exactly is going on. Additionally in the VM we do get the same warning as stated above, on the line with the issue so that helps to track it down easy enough. Well lets go back to the dartboard and enable checked_mode:
wat1
Failed type check: type String is not assignable to type Number

Well that's kind of a reasonable error. Must get the same in the Dart VM using the 'enable_type_checks' flag.
wat1
Unhandled exception:
'bootstrap_impl': Failed type check: line 1468 pos 22: type 'OneByteString' is not assignable to type 'num' of 'other'.
 0. Function: 'IntegerImplementation.+' url: 'bootstrap_impl' line:1468 col:22
 1. Function: '::main' url: '/home/mbutler/dart/Test/Test.dart' line:3 col:11

Okay WAT? That error is almost as long as the first one but its a little more obscure I think. I see from the very last line of the error where the issue is in my code (I notice this now in the first error as well. I'm so used to seeing my code point at the top of the error with subsequent library calls following after that, that I didn't realize to look at the bottom of the error for my error). But the error itself is interesting "OneByteString" is not assignable to type "num" of "other"...? Of other what? String not assignable to num is understandable though not necessarily expected. But of other? I'm not sure. Just rambling a little bit at this point as well as I procrastinate getting some work done in the office today.

I'll post again later today if I get the opportunity as Seth has a couple of other blog posts written that I want to follow up on. Particularly on his one about 'this'.

Friday, January 20, 2012

Not So Generic

I was working on a post on Generics in Dart, had a significant amount written yesterday but was running into a couple of bugs (bug reports were, for the most part, previously submitted). Coming back today and looking at my post and trying to pick up where I left off, I decided the best option, for your sake and mine, was to delete it. I had a mess of things jumping from one topic to another, code samples that had far too much fluff and useless elements that it was far from clean, concise and illuminating. So I admitted my mind wasn't as focused as it should have been and deleted the post entirely. It wouldn't serve much purpose to try and clean it up at all.

I've decided for the time being I'm going to leave the generics to Seth Ladd's most capable post: Generics in Dart. I highly recommend reading it and pay particular attention towards the end of the post to help avoid some 'gotchas'. He does a great job explaining covariance and keeping it simple at the same time. The only additional thing I will add is that not only can you use generics on the initialization of a variable but also in the declaration as a type. Doing so will really help, particularly the editor, to keep a close eye on what you are assigning or adding to a variable. The only example I will show is very brief: http://try.dartlang.org/s/lywo
main() {
  var list1 = ["One", "Two", "Three"];
  List list2 = new List();
  List list3 = ["One", "Two", "Three"];
  
  list1.add(4); // No warning because the type isn't specifically declared.
  list2.add(4); // generates a warning
  list3.add(4); // Generates a warning
}

Assuming you are not running in checked mode, all three cases will run properly. However, list2 and list3 will generate errors on the add lines indicating that int is not assignable to a string. The first case, list1 will quietly and happily accept 4 as being added to it, even though we initially specified that the variable of Dynamic type list1 is a List of type Strings. If running in checked mode, all three will fail.

Apart from that, I decided that over the weekend I'm going to start working on an introductory tutorial to Dart. Something for the programmer with little experience. Many of the resources I've seen have been geared towards those experienced with Java programming or heavy JavaScript users. I'm debating if I should go with just VM initially and get the foundations down or just DOM initially so they can get right into results. Because lets face it, when you first start as a programmer we're all about the results and making the computer do something cool rather than all the theory and foundations. However I believe I can offer that with the VM on a starting level.

Thursday, January 19, 2012

Mapping the Future

So yesterday I looked at Lists. Today, I'm onto Maps. As has been the way, this is based off of the information I read from Seth Ladd's Blog series on Dart. Maps and Hashes in Dart.

Maps are a great utility in web programming especially. They are an efficient way to create a relationship between a 'key' to a value. In Dart values can be of any type, and similar to Lists, they do not have to be of the same type. That is a map can contain keys which are strings, numbers, and other classes including Lists etc.

Unlike some other languages though (Ruby for instance), the 'key' in a map must be a string. It cannot be an instance of a class for example. You can't use a number as a key (which would cause confusion with Lists). 99.99998% of the time this won't be a problem as there is only a very rare case you would need anything other than a string. I will admit though I did enjoy the lighter-weight use of 'symbols' in Ruby, particularly for maps. But different roles for the languages. [edit:] I see a note at the bottom of Seth Ladd's blog indicating that this may change in the future. [/edit]

In Dart, as in most (but not all other languages), when you query a map for a key that doesn't exist, null is returned. I much prefer this over the alternative of a runtime 'indexOutOfRange' error as I have seen in another language. So for instance if we have the following:
main() {
  var map = { "Hello" : "World" };
  print(map['Foo']);
}

The result is:
null

We can then do a simple check to see if the value is null and assign it if we have to. Its also extremely easy to add to a map. Even if the key doesn't exist, if we assign to it, it will be added. So for instance we can do the following:
main() {
  var map = { "Hello" : "World" };
  map["Hello"] = "Joe";
  map["foo"] = "bar";
  // Map now contains { "Hello" : "Joe", "foo" : "bar" }
}

So that's nothing new to people familiar with Maps from other languages and with those two methods we have an easy way to check for a value and assign one to it if it doesn't exist otherwise. However there is an interesting function that exists in Dart for Maps: putIfAbsent.
This function allows us to query a map for a key and returns its value. However if the key was not previously defined, it will call the function you pass to it and assign the return value to that key. Lets look at an example:
main() {
  var map = { "Hello" : "world" };
  var res = map.putIfAbsent('foo', assign() {
      // Do some stuff
      return 'bar';
    });
  print(res);
  print(map['foo']);
}
bar
bar

So as you can see map is queried for the key 'foo'. When it does not find it, it then calls the function (in this case I named it assign). You can write some code to calculate a value and then whatever is returned is the value that is assigned to that key in map. That value is then also returned by map.putIfAbsent and assigned to the res variable. Something important to note however: putIfAbsent checks if the key is absent, not the value of that key. That means if the key has already been assigned but is simply null, the method will return null and not call the function that you pass to it. As the following demonstrates:
main() {
  var map = { "Hello" : "world", "foo" : null };
  var res = map.putIfAbsent('foo', () => 'Bar');
  print(res);
  print(map['foo']);
}
null
null

This is the same as the containsKey method. The method verifies if the key exists in the map already, not if it has a non-null value.

Another interesting tidbit is if you want to run in checked mode, or at least generate warnings when working with maps, you have option to specify the types to be expected. As previously mentioned, at the moment the key must be a string, but the capability, using Generics, allows you to catch potential errors or surprises. Seth does a great job explaining the use of generics for Maps on his blog, so check it out there.

Wednesday, January 18, 2012

A List of List stuff

So I continued on reading Seth Ladd's blog post on Lists and Arrays in Dart. One thing I didn't see mentioned that I think should be said, is that Lists are not type dependent. For people familiar with current dynamic languages such as Python, Ruby, and JavaScript that's not a big deal. But for those coming from Java, C, and other similar languages it is important to note that a List doesn't need to be all the same type. For instance the following is entirely valid:
List list = [1, "two", 3.48283008, [4, 4, "four", 4]];

So a List can contact mixed variable types, numbers, strings, even more Lists. However for those familiar with Javascript and other such dynamic languages who are used to associative arrays, its important to know that Lists are not that. As such you cannot assign a value to an index that does not yet exist.
List list = new List();
list[0] = "zero";
list[1] = 1;
IndexOutOfRangeException: 0

However, for extendable arrays, you can call set on length and increase the value. So you can grow, or shrink with more than just the add or addAll methods. For instance the following is valid:
List list = new List();
list.length = list.length + 2; // Now have two null values in the list.
list[0] = "zero";
list[1] = 1;
list.length = list.length - 1; // This removes the last value from the list.

Something important to note however, currently the API does not specifically state that changing the length to a value lower than the current will always have this effect. Currently there is no guarantee that the values will be removed from the end of the List, or even that they will be removed at all. I can see a situation where, if the values at the end are not null, then they may not reduce the size of the List. That or when setting list size, the new value must be greater than or equal to current size. As the libraries are improved and change (and there will be change don't doubt that for a second), we'll get a better idea of what to expect.

There are a number of List methods which can be extremely useful, that require a function be passed to them as an argument. You can create these functions inline or pass the name of another function. I'll cover more on passing functions in another post. Some of these methods are: filter, some, every, sort. Some and Every return true or false values. every returns true if every value in a list meets the criteria defined by the function passed as an argument. some returns true if at least one of the values meets the criteria.
bool isNum(var e) => e is num;

main() {
  var list = [1, 2, "three"];
  print( list.every(isNum) );
  print( list.some(isNum) );
}

Gives the result:
false
true

Something to note is that the some, every and filter methods are defined as part of the Collection interface not specifically in List.

Something to make note of is that filter will return a new list of items that return true from the passed function, whereas the sort method will modify the existing list rather than return a new one. Take the following code as an example of how to use each.
main() {
  var list = [4, 3, 6, 1, 2, 5];
  list.sort(compare(a,b) {
    if (a == b) {
      return 0;
    } else if (a > b) {
      return 1;
    } else {
      return -1;
    }
  });
  // List is now [1, 2, 3, 4, 5, 6]

  var evens = list.filter(f(e) => e % 2 == 0);
  // list is still [1, 2, 3, 4, 5, 6]
  // evens is [2, 4, 6]
}

In the above code the list.sort method modifies the List we're working with, whereas the list.filter method returns a new List and does not make any changes to the existing list. One thing I'm not currently sure of is why the difference. Is it that sort is not removing or modifying the elements of the list where as filter does? To me anything that modifies the existing List, even if it's changing the ordering of the elements, should return a new List, which I can then assign back to the same variable if I so choose. As it stands now, I have to make a copy of the list prior to the sort if I want to keep the original ordering.

Another item to note, based on my limited trials, when iterating over the values of a List, elements are passed as value not reference. Thus if you make any changes to the value, they will not persist outside of the scope of the function. Here's an example of what I mean:
main() {
  var list = [1, 2, 3];
  for(var x in list) {
    x = x*2;
  }
  // List is still [1, 2, 3]
  
  list.forEach(f(x) { x = x*2; });
  // List is still [1, 2, 3]
}

In the first case, using the for loop I can understand that the elements are passed as values and that it does not modify the original list. However in the second case, using the forEach method, you would expect that any changes to the value there would be reflected in the list after the fact.

In any case. I highly recommend reading through the API references for List and for Collection to get an idea as to what other capabilities they hold.

A List of Errors.

Before I get started on this post on Lists (actually, I already started it yesterday but didn't complete it yet ;), I did want to share the information that Seth Ladd recently shared regarding the Dart issue tracker. Sometimes it can be hard to locate or you just can't recall the location to access the tracker. Well never fear! There is a new redirect URL available to use.
dartbug.com
Redirects to the main issue list.
dartbug.com/new
Redirects to the new issue template.
dartbug.com/1199
Redirects to a specific issue number (in this case 1199, the one I submitted and mentioned in my last blog post).
So I'm continuing with Seth Ladd's series on Dart. Today I'm looking at Lists and Arrays. One particular quote from the post particularly grabbed my attention and interest:
Fun fact: As of 2011-12-22, the word "array" does not appear in the Dart spec.'
This interested me as virtually all languages have arrays and refer to them as such. But sure enough a quick search through the current language spec reveals no results for 'array'. Now, as stated in Seth's blog, Dart arrays are Lists. I found it an interesting point none the less.

Interestingly enough however, the VM does output an error which refers to arrays. But when compiled to Javascript (via frog or dartc) the errors properly reflects 'lists'. First I'll demonstrate the code and error then I'll actually provide some additional comments regarding the code.
main() {
  var list = new List(3); // Create an immutable, or 'non-extendable' list
  list.add(4); // Now try to change it (extend it).
}

This results in the following error:
Unhandled exception:
UnsupportedOperationException: Cannot add to a non-extendable array

But when executed on the dartboard we get the following error:
UnsupportedOperationException: Cannot add to a non-extendable list

It surprises me a little that the VM does specifically reference array whereas the code compiled to Javascript, which does have arrays, correctly references a list. There already exists a couple of bug reports. One which I'll only briefly mention, indicates that the word 'array' should be removed from all internal libraries. The second, which I added a comment to regarding the VM specifically is: Issue 1028. This issue aims to be more encompassing removing references from the Libraries as all references which may become visible.

So you may be wondering why it is that List provides an add method if it just generates an error. Well that's because there are two types of lists. A 'non-extendable' list which is created by passing a size to the constructor, or you can create a list without the optional argument to the constructor which creates an empty list you can extend by adding (or removing) values from. An extendable list can also be created from a list literally such as:
var list = [1, 2, 3];

Note however, that unlike some other languages (such as Ruby), you can not call a method on the List literal. Thus the following is not valid code:
var list = [1, 2, 3].add(4);

You'll receive a warning indicating "expression does not yield a value". But it will compile and run (at least until you try to call a List method on list such as length which will generate a runtime error.

However there is a bug in the add method which I have just discovered. The API for List.add indicates that the return type is 'void' (or no return value). This works as expected in the dartboard (javascript). However in the Dart VM, list.add(value) returns the new size of the list. I have submitted a bug report for it here: Issue 1213.