Dart MUD... Progress 1

So before I get started, I just want to put out there that the latest integration build of DartEditor has been released, and you should pick it up here. This build contains the latest IDE, the SDK and now Dartium as well! For those of you running 64-bit systems, it is a 32-bit build of Chromium so you may encounter some library errors if you have not installed the extra components you need.

So now that that news is out of the way.. a little progress report on my Dart MUD. (I may need to come up with a better development name, as it conflicts with the already existing MUD: Dartmud.com) So far I have a minimal server, from which I've managed to remove most unnecessary code from and make the mudlib independent of the server. I have a generic MudLib object which handles most interactions between server and specific mudlib components. A User object which handles the client connections and various user properties (username, passwords, etc). However I'm not completely happy having the socket implementation directly part of the User object, and may break down into a connection object which the User class will either have it as a property, or may inherit from (just because someone connects, doesn't mean it will result in a User on the system).

After some playing around with a static class which contained the various commands a player can execute, I decided to instead create a small class, and each command is an instance of the class. All of which are managed by a CommandDaemon/Command Manager. I can see this allowing for more flexibility in adding commands globally and perhaps even in just specific rooms or objects. On the note of Rooms and GameObjects, I have classes for these also implemented, as well as a 'container' interface and a factory class for it. Users and Rooms both currently extend the factory class for the container.

At this point, I've added a few 'commands', including the exit and shutdown commands from the EchoServer (though now setup to be used with the Command Manager I have implemented). I've also added a basic 'look', 'broadcast' and 'help' command, and am in the process of fleshing out these respective commands so they properly handle any arguments sent to them (broadcast works to notify all users, regardless of if they are in the same room as the person issuing the commands.

I still have so much to do. Currently when connecting you are prompted for a username/password to log in. However you can enter anything an be granted access with the username you enter. I need to be able to create a new user, to save the user, and to load them when logging in. Also I have a very basic room, but only the one. I still have to create more rooms and implement travel commands to move between rooms. To say nothing of the other normal things you would expect on a mud including equipment, objects, money, combat, etc etc etc.

While I still have a very long way to go to make a basic mud, I'm quite pleased with the progress I have been making towards that end. Something I may want to look towards as well is some type of 'reload' or 'refresh' command, as I have to shutdown the server and restart it after each change I make. I'll also need to work on a memory management daemon of some kind too, so that once the mud gets to a decent size, objects will only be loaded/created when accessed, and after a period of time, removed if not used or near an active player.

One of these days I'm going to make an effort to learn git/git hub too.

Featureless Sockets

Okay, so the title is a bit of a stretch. Dart Sockets are far from featureless. As some of you may or may not know, I've decided to take a segue in my EchoServer towards working on a basic MUD and MUDLib written in Dart. It will be far from full featured, and far from fully functional but it will be the basis and a nice little start.

Within a few minutes of converting the EchoServer to be the basis of the MUD driver, I've already run into an issue of a missing feature with Sockets in Dart. I'm unable to get the host of an incoming connection. That is, I can't find out who is connecting to me. While it's a trivial matter for a MUD, or many other services, one would expect such functionality for various uses including HTTP server logging, etc.

After digging through the bleeding edge to see if it may have been recently included but just not yet updated in the API, I still can't find any reference to such properties. So as such, I've created a bug report over in the Dart issue tracker. If you're interested, or want to star the report you can view it here at: Issue 1819

As I progress with the Dart core and MUDLib I'm sure I'll find some additional bugs to report, or just missing features I can think could be added. Additionally I'll keep progress reports available on here and eventually release the source on GitHub or something similar.

Serving Sockets... The Salad

I haven't had the opportunity to do much with Dart lately, due to scheduling. However this afternoon I had a little chance to write some more. I decided that I wanted to modify the EchoServer to maintain the connection and close the connection only when receiving a specific command. I also wanted to add a command to shutdown the server itself completely. In order to properly accommodate this, I needed to refactor the server code a little. I wrapped the ServerSocket in a manager class:

#import('dart:io');

class ServerManager {
  ServerSocket _listenServer;
  
  ServerManager() {
    _listenServer = new ServerSocket("127.0.0.1", 5700, 0);
    
    _listenServer.connectionHandler = this._handleConn;
  }

  void _handleConn(Socket conn) {
    StringInputStream clientIn = new StringInputStream(conn.inputStream);
    
    clientIn.lineHandler = () {
      String input = clientIn.readLine();
      print("Received: $input");
      if(input.toLowerCase() == 'stop') {
        String cls = "** Stopping Server. Closing connection **\n";
        conn.writeList(cls.charCodes(), 0, cls.length);
        conn.close();
        _listenServer.close();
        print("** Stopping Server! **");
      } else if(input.toLowerCase() == 'exit') {
        String cls = "** Closing connection to client **\n";
        conn.writeList(cls.charCodes(), 0, cls.length);
        conn.close();
        print("** Closing connection to client. **");
      } else {
        String output = "${input.toUpperCase()}\n";
        conn.writeList(output.charCodes(), 0, output.length);
        print("Sent: $output");
      }
    };
    
  }
}

void main() {
  ServerManager sMan = new ServerManager();
 
}

So as you can see I made a few changes from my original EchoServer. As mentioned above I wrapped the server in a manager class, this enables me to easily close the server socket without using a global variable. In addition I added a couple clauses to check for the 'stop' or 'exit' commands which will stop the server or just close the client connection respectively. And finally I stopped pulling the output stream of the sockets directly, and instead use the writeList methods directly on the socket itself. I wasn't gaining any real benefit by creating an additional variable for the socket's OutputStream, so I just dropped it altogether.

Now as is, the above will run and accept connections and echo any new lines until the stop or exit commands are received. If the exit command is received, then the server will close the connection to that socket. If stop is received it will close the connection to that socket and then tell the server to close. However because of the event driven nature of the server, the Sockets and ServerSockets are not blocking. That is, even without adding any additional threads (Isolates), we can accept connections from multiple sources. If you open up multiple telnet connections to the host, you can see how you can send data and receive responses on each connection independent of the other.

But this also leaves us with a small issue. If we tell the server to stop from one telnet session while the other is still active.. the server will accept the stop command, and it will schedule the ServerSocket to be stopped, but not until the other socket has been closed. Try it out and you will see that the connection in which we issue the stop command is disconnected, and the console will indicate that the server is stopping. But the other telnet session will remain active until we issue an exit or stop command. Only once the 2nd session is closed will the server stop. And if for some reason the other session does not terminate properly (for instance connection drops or the telnet application is closed before issuing an exit/stop command) then the server will hang, not accepting new connections but not terminating either (assuming eventually the socket will time out but potentially not since I do not have those error handlers in place either).

This may be the desired situation with some servers to shut them down gracefully for instance, however in our EchoServer we want it to shut down immediately if it receives the stop command. So we'll need to keep a list of active connections and iterate through them and close each one, then stop the server. So I ended up with the following:

#import('dart:io');

class ServerManager {
  ServerSocket _listenServer;
  List _socketList;
  
  ServerManager() {
    _socketList = new List();
    _listenServer = new ServerSocket("127.0.0.1", 5700, 0);
    
    _listenServer.connectionHandler = this._handleConn;
  }
  
  void sendStops() {
    List cls = "** Server received stop request. Closing connection to client **\n".charCodes();
    
    while(!_socketList.isEmpty()) {
      Socket conn = _socketList.removeLast();
      conn.writeList(cls, 0, cls.length);
      conn.close();
    }
  }
  
  void _handleConn(Socket conn) {
    _socketList.add(conn);
    StringInputStream clientIn = new StringInputStream(conn.inputStream);
    
    clientIn.lineHandler = () {
      String input = clientIn.readLine();
      print("Received: $input");
      
      if(input.toLowerCase() == 'stop') {
        sendStops();
        _listenServer.close();
        print("** Stopping Server! **");
      } else if(input.toLowerCase() == 'exit') {
        String cls = "** Closing connection to client **\n";
        conn.writeList(cls.charCodes(), 0, cls.length);
        int sockInd = _socketList.indexOf(conn);
        _socketList.removeRange(sockInd, 1);
        conn.close();
        print("** Closing connection to client: $sockInd **");
      } else {
        String output = "${input.toUpperCase()}\n";
        conn.writeList(output.charCodes(), 0, output.length);
        print("Sent: $output");
      }
    };
    
  }
}

void main() {
  ServerManager sMan = new ServerManager();
 
}

As you can see I also added a method sendStops just to iterate through all the sockets, popping them out of list and sending them the stop notice and disconnecting them. I made this separate from the actual stopping of the server in case it should ever be required for any other reason as well. Initially I tried using a Set to hold just unique connections, and provide easier way of removing elements however I found out that there's an issue with Set's in that any values stored in a set must implement Hashable. This wasn't added in the API documentation and it was only after a little digging through the DartBug page and Newsgroup that I found this is 'expected' behaviour. As such, I had to use the list. For a specific 'exit' command I have to get the index of the value and remove it from the list with removeRange with a size of 1 element. I also setup the broadcast message directly to a List of Int's immediately just to avoid having to convert it multiple times as I iterate through the connections. While still missing any error handling, etc. I'm rather pleased with how the server is progressing and in some ways it conjures up images of the old school MUD's. Maybe a project to play with?

Serving Sockets... The Soup Crackers

So as I mentioned in my previous post, there is more than one way to communicate through sockets. Below I have show 3 different, though similar ways to make use of sockets. This is far from exhaustive and please note that this is for demonstration purposes only and as is, should not be used in production code of any kind. They're missing error checking, stream verification, etc.

I am interested if any one has any comments or feedback as to use cases where they would choose to use one of the particular methods over another. Everything below is fairly procedural and could probably be implemented much cleaner as Objects, but as mentioned this is purely a test case using the EchoServer I wrote a couple posts back.

#import("dart:io");

void main() {
  sockets_with_handlers();
  
  sockets_with_direct_streams();

  sockets_with_socket_streams();
}

void sockets_with_handlers() {
  Socket usingHandlers = new Socket("127.0.0.1", 5700);
  String test = "SocketHandlers Test String\n";
  
  // Called when we successfully connect
  usingHandlers.connectHandler = () {
    print("Handers: Connected");
  };
  
  // Called when we can write to the socket.
  usingHandlers.writeHandler = () {
    usingHandlers.writeList(test.charCodes(), 0, test.length);
    print("Handers Sent: $test");
  };
  
  // Called when we receive from the socket.
  usingHandlers.dataHandler = () {
    int availBytes = usingHandlers.available();
    List buff = new List(availBytes);
    usingHandlers.readList(buff, 0, availBytes);
    print("Handers Received: ${new String.fromCharCodes(buff)}");
  };
  
  // Called when the input stream from socket is closed.
  usingHandlers.closeHandler = () {
    print("Handers: End of stream. Closing connection");
    usingHandlers.close();
  };
}

void sockets_with_direct_streams() {
  Socket usingStreams = new Socket("127.0.0.1", 5700);
  String test = "Direct Socket Streams test string\n";
  
  // To be 'pure' without the handlers we assume connection is successful.
  // NOTE: Not recommended!
  print("Direct Streams: Connected");
  // Wrap the input stream in StringInputStream so we can
  // make use of the convenience functions.
  StringInputStream inStream = new StringInputStream(usingStreams.inputStream);
  OutputStream outStream = usingStreams.outputStream;
  
  // Write to our stream
  outStream.write(test.charCodes());
  print("Direct Streams Sent: $test");
  
  // Use our handle wrapper to read lines
  // Saves us from using dataHandler directly
  inStream.lineHandler = () {
    String input = inStream.readLine();
    print("Direct Streams Received: $input");
  };
  
  // All bytes have been read and input stream is closed.
  inStream.closeHandler = () {
    print("Direct Streams: End of stream. Closing connection");
    usingStreams.close();
  };
}

void sockets_with_socket_streams() {
  Socket usingSocketStreams = new Socket("127.0.0.1", 5700);
  String test = "Socket In/Out Streams test string\n";
  
  // To be 'pure' without the handlers we assume connection is successful.
  // NOTE: Not recommended!
  print("Socket Streams: Connected");
  
  SocketInputStream inStream = new SocketInputStream(usingSocketStreams);
  SocketOutputStream outStream = new SocketOutputStream(usingSocketStreams);
  
  // Write to our stream.
  outStream.write(test.charCodes());
  print("Socket Streams Sent: $test");
  
  // Called when new data arrives in our SocketInputStream
  inStream.dataHandler = () {
    List buff = inStream.read();
    print("Socket Streams received: ${new String.fromCharCodes(buff)}");
  };
  
  // Input stream has been closed. Make sure output stream
  // and socket itself are also closed.
  inStream.closeHandler = () {
    print("Socket Streams: End of stream. Closing connection");
    outStream.close();
    usingSocketStreams.close();
  };
}

Direct Streams: Connected
Direct Streams Sent: Direct Socket Streams test string

Socket Streams: Connected
Socket Streams Sent: Socket In/Out Streams test string

Handers: Connected
Handers Sent: SocketHandlers Test String

Direct Streams Received: DIRECT SOCKET STREAMS TEST STRING
Socket Streams received: SOCKET IN/OUT STREAMS TEST STRING

Handers Received: SOCKETHANDLERS TEST STRING

Direct Streams: End of stream. Closing connection
Socket Streams: End of stream. Closing connection
Handers: End of stream. Closing connection

One of the first things you'll notice with the output is that due to the non-blocking nature of the callbacks, some of the calls happened in near-parallel. In the sockets_with_socket_streams function, while I didn't specifically make use of it, we could have also wrapped the SocketInputStream with a StringInputStream as well, similar to what we did in the sockets_with_direct_streams function.

Serving Sockets... The Soup

Alright, first thing first. Announced during yesterday's Episode 2 of Dartisans Hangout, Dartium binaries are being relased. Initially for Mac OS X and Linux with Windows binaries to follow soon. I won't go on about this much as it has already been mentioned again and again, and again.

One thing to note however, is that currently the Linux Build is a 32-bit version so if you're running a 64-bit OS you will need to ensure you have the proper 32-bit libraries downloaded and installed. Or alternatively, wait for the 64-Bit builds to arrive.

Continuing with my small series on Sockets in the Dart:IO library, I thought well I have an EchoServer written, albeit extremely primitive, my next 'logical' step is to make a client which connects to the server, sends a string, receives the echo response and disconnects. This should be pretty straight forward, since it was so easy to make the server... Oh boy was I ever wrong about that. Not that Sockets are overly complex or anything, they just have many more ways of accomplishing the same thing, but they can't be used in conjunction as I found out with the following:

#import("dart:io");

void main() {
  // Create a new socket connecting to localhost and port 5700
  // the same port as our echo server we wrote is running on.
  Socket conn = new Socket("127.0.0.1", 5700);
  StringInputStream inputStr;
  OutputStream outputStream;
  
  // method is called when connection is established.
  conn.connectHandler = () {
    print("Now Connected");
    inputStr = new StringInputStream(conn.inputStream);
    outputStream = conn.outputStream;
    
    String test = "This is a simple test\n";
    outputStream.write(test.charCodes());
    print("Sending: $test");
    
    // We wrapped the input stream for this easier reading
    // using the lineHandler as opposed to bulk data.
    inputStr.lineHandler = () {
      String input = inputStr.readLine();
      print("Recieved: $input\n");
    };
  };
  
  // Called when the last byte of data has been read from the socket.
  // Socket potentially still open for writing.
  conn.closeHandler = () {
    print("Connection closed. Last byte of data has been read from stream");
    conn.close();
  };
  
}

Now Connected
Unhandled exception:
StreamException: Cannot get input stream when socket handlers are used
 0. Function: '_Socket@14117cc4.get:inputStream' url: 'dart:io' line:4231 col:9
 1. Function: '::function' url: 'src/dart/echo/EchoClient.dart' line:13 col:54
 2. Function: '_Socket@14117cc4.firstWriteHandler' url: 'dart:io' line:4274 col:64
 3. Function: '_SocketBase@14117cc4._multiplex@14117cc4' url: 'dart:io' line:3898 col:23
 4. Function: '_SocketBase@14117cc4.function' url: 'dart:io' line:3996 col:59
 5. Function: 'ReceivePortImpl._handleMessage@924b4b8' url: 'bootstrap_impl' line:1734 col:22

So as we can see, this didn't work quite as cleanly as we'd have liked. Apparently the issue above is with my mixing the Socket handlers, such as connectHandler and closeHandler, while also pulling the socket's input and output streams themselves. In the case of the InputStream wrapping it in a StringInputStream handler to make dealing with input a little easier since we only expect text.

Initially I thought this error was telling me that I cannot use any socket handlers, if I wanted to pull the IO streams directly from the Socket. However I only just now stumbled upon an easier solution. The error isn't with the use of connectHandler and closeHandler on the socket. Apparently only the closeHandler is triggering the error. So I can move that method to be called on the StringInputStream instead. So a quick re-write gives me the following:

#import("dart:io");

void main() {
  // Create a new socket connecting to localhost and port 5700
  // the same port as our echo server we wrote is running on.
  Socket conn = new Socket("127.0.0.1", 5700);
  StringInputStream inputStr;
  OutputStream outputStream;
  
  // method is called when connection is established.
  conn.connectHandler = () {
    print("Now Connected");
    inputStr = new StringInputStream(conn.inputStream);
    outputStream = conn.outputStream;
    
    String test = "This is a simple test\n";
    outputStream.write(test.charCodes());
    print("Sending: $test");
    
    // We wrapped the input stream for this easier reading
    // using the lineHandler as opposed to bulk data.
    inputStr.lineHandler = () {
      String input = inputStr.readLine();
      print("Recieved: $input\n");
    };

    // Called when the last byte of data has been read from the socket.
    // Socket potentially still open for writing.
    inputStr.closeHandler = () {
      print("Connection closed. Last byte of data has been read from stream");
      conn.close();
    };
  };
  
}

Now Connected
Sending: This is a simple test

Recieved: THIS IS A SIMPLE TEST

Connection closed. Last byte of data has been read from stream

Well then, that's working so much better. And in fact I could leave it here.. and will for the time being. However later, perhaps tomorrow, I will write a continuation of this post on some of the other methods of using and communicating with sockets. They are plentiful.

Servering Sockets... The Appetizer

So I decided it was time to start playing around with some Sockets. As we already know, I'm not a huge fan of the chat sample that's available, as I find it a little too much. Fortunately the Sockets and ServerSockets are not too different from the Process API, so keeping that in mind I wrote a dead simple echo server. This does not adhere to any protocols or anything of that nature. It just accepts input from a client and sends it back to the client. Truth be told, I should be a little ashamed of the following code as its far from complete, far from safe, and far from extendable. But it was a good starting point and I'll be able to now take it and re-factor it convert it into something usable. The goal was simple, to write a simple server and have evidence that communication was bi-directional.

So first I'll provide the code and then a little discussion about it.

#import('dart:io');

void main() {
  // Create a server and bind it to an address and port number.
  ServerSocket listenServer = new ServerSocket("127.0.0.1", 5700, 0);
  
  // This is called when a connection is received.
  listenServer.connectionHandler = handleConn;
}

// Socket conn is the incoming connection.
void handleConn(Socket conn) {
  // Wrap the input from the connection around a StringInputStream to make reading it easier.
  StringInputStream clientIn = new StringInputStream(conn.inputStream);
  // Get the output stream too so we can write back to the client.
  OutputStream clientOut = conn.outputStream;
  
  // Line handler is called when a new line is received to the StringInputStream
  // which we've wrapped around the InputStream from the socket.
  clientIn.lineHandler = () {
    String input = clientIn.readLine();
    print("Received: $input");
    String output = "${input.toUpperCase()}\n"; 
    clientOut.write(output.charCodes());
    print("Sent: $output");
    conn.close();
  };
  
}

So as you can see above the code is very short and simple. I recommend running this in the IDE as currently the process will run forever. Later on I may add a command to shutdown the server from the client. To test out the above code start the server with the DartVM then open a telnet session. Telnet to 'localhost:5700' Then simply type a line of text and hit enter. You should get the same line of text sent back to you but uppercase.

Basically we create the server, and then just handle any incoming connections with it. The connectionHandler is called when a new connection is established to the server, and the function is passed the Socket to the incoming connection.
From that socket we get the InputStream and OutputStreams. For the InputStream we wrap it in a StringInputStream which simplifies reading strings from the connection, and since we're not really worried about receiving any bulk or binary data as such.

Next, thanks to the StringInputStream, we setup a lineHandler which is called when a new line of text is received by the stream. Then we use readLine to store that line in a variable.

We print what we've received to the console, then create a new string which is uppercase version of the line we just received, and add a new line character to the end of it. We want to add the newLine character because when we use the readLine method, it strips the new line character for us automatically. In this case I wanted to return an uppercase string of the original just for visual confirmation that what is being returned is different in some way from the original so it can't be blamed as a local echo by the telnet client or anything like that.

Next we write the output back to the output stream. Because the OutputStream is not wrapped by a String stream we need to convert our string into a list of character codes which are then in turn sent back through the SocketOutputStream. We also write to the console what we sent.

Finally we close the socket. In a later example we may leave this open until we send a specific comment to terminate the client connection, or specifically have the client close the connection instead.

As I mentioned, this example is pretty much as simple as you can get. It also is missing any error handling, any protocol conformation, but it gave me a comfortable start/introduction to sockets in general. If you have any suggestions or comments regarding the code by all means I eagerly look for feedback or suggestions.

String that Process

So a comment from Florian Loitsch on my previous post indicated that as opposed to creating a new string from character codes as I was doing for handling output of the process, I can use the existing StringInputStream to automatically wrap the basic input stream into a string. As indicated by the API I can then pull the strings on a per-line basis or as a string chunk. I've revised my example to put both methods to use. Before commenting on it, lets see the code:

#import("dart:io");

void main() {
  // Create a new process with list of arguments and start it.
  Process proc = new Process.start('/usr/bin/snmpget', ['-v', '2c', '-c', 'public', '127.0.0.1', '.1.3.6.1.2.1.1.1.0']);
  
  // Once started, handle the process including receiving input.
  proc.startHandler = beginHandle() {
    print("Inside of start Handler");
    
    // Create StringInputStreams from the basic input streams of the process.
    StringInputStream res = new StringInputStream(proc.stdout);
    StringInputStream err = new StringInputStream(proc.stderr);
    
    // A handle a new line of text added to the input stream (stdout).
    // readLine does not include the terminating new-line character
    res.lineHandler = showLine() {
      print("Inside InputStringStream lineHandler");
      print(res.readLine());
    };
    
    // Handle new data sent to the input stream (stderr)
    // This happens whenever any data is available and does not
    // wait for a new line.
    err.dataHandler = errRead() {
      print("Inside InputStringStream dataHandler");
      print(err.read());
    };
  }; // End off Proc.startHandler
  
  // When the process has ended this handler is called
  // passes process's exit code.
  proc.exitHandler = exitHandle(int exitCode) {
    print("Inside of exit Handler");
    print("Exit code is: ${exitCode}");
    proc.close();
  };
  
  // Called when an error occurs when spawning the process.
  proc.errorHandler = errorHandle(ProcessException errorMsg) {
    print("Inside of Error Handler");
    print(errorMsg);
  };
}

I've also added some comments to the above to help clear it up a little as well. With the particular process I'm launching generally I'm only interested in parsing one return line at a time. So as such with my StringInputStream for stdout I decided to use the lineHandler method. Because stderr may produce more than one line of error, I used the dataHandler trigger the reads. Note that the two are mutually exclusive. That is the same StringInputStream can only define a lineHandler or a dataHandler. Not both.

The lineHandler is called each time a new line is available to the stream. dataHandler is called anytime there's new data available, no matter if it contains a new line or not. This is good if you want to receive a bulk of a string, or potentially expect an unterminated string.

Note I also did not methods for the errorHandlers in this simple example, nor verify if the stream is currently open. Using both this method and the previous one of directly using the basic InputStreams are roughly the same size, however unless you explicitly require the raw character codes, I would personally suggest using the StringInputStream over the basic InputStream for potential performance gains and to help future-proof your code against any internal library chances which make affect results or performance down the road with the InputStream and string from character codes.