Parabolic Logic

The world isn't flat..Software shouldn't be flat either

You are here: Home - Coding

Category Archives: Coding

Scripting languages part 2

Community support

DevOps, cloud deployment, test- driven development and continuous integration – the demands on a sysadmin change and evolve, but the requirement to learn something new is constant. Everyone uses Bash to some extent but, you’ll need to learn Bash plus one other. Perl was the traditional Swiss Army chainsaw of Unix admins through the ‘80s and ‘90s, gradually losing ground to Python and then Ruby over the last decade or so.

Anyone who started work in the ‘90s or earlier will be comfortable with it, so finding someone to help with your scripts is often not a problem. However, the world doesn’t stand still, and many tech businesses have standardised on Python, which is used extensively at Google, for example.

Much of the software necessary for modern sysadmin work is Python based although the same can be said of Ruby. Ruby benefits from being the basis of Chef and Puppet, as well as Vagrant and Travis CI, meaning a little familiarity will be helpful anywhere that uses them for deployment.

The web frameworks and testing tools written in Ruby have popularised the language at many of the younger web companies. NewLISP has a much smaller community supporting it, and there aren’t many ready made solutions and you may know no-one who uses it. The keenness of the online community goes some way to ameliorate this deficiency, but you have to ask who will maintain your tools when you leave a company?


Before reaching 1,000 lines of code, Bash scripts become unmanageable. Despite its procedural nature, there are attempts to make an object-orientated (OO) Bash .

There’s more than one library for that – CPAN is a useful resource for Perl.

We don’t recommend it, we think it’s better to modularise. Functional programming (FP) in Bash (http://bit. ly/BashFunsh) is also impractical. Perl’s bolted on OO won’t be to everyone’s taste, but does the job. Perl has fully functional closures, and despite syntactical issues, can be persuaded into FP – just don’t expect it to be pretty. For that you should wait for Perl 6.

Python is equally happy with imperative, OO and also manages FP. Functions are first class objects but other features are lacking, even if its list comprehension is very good. Mochi, the FP language (, uses an interpreter written in Python 3. Ruby is designed as a pure OO language, and is perhaps the best since Smalltalk. It can also be persuaded to support a functional style of programming.

But to get FP code out of Ruby, you’ll have to go so far from best practices that you should be using another language entirely. This brings us neatly to NewLISP, an elegant and powerful language with all the functional features at your fingertips. NewLISP uses a pseudo OO implementation in the form of functional-object oriented programming (FOOP), but this doesn’t mean, however, that it can cut it for real OO programming.

Extending the language

None of these scripting languages are as bloated with classes as, for example, Java so that you’ll need to use non-core libraries (or modules as they are sometimes called) for writing many scripts. How comprehensive these are, and how easy they are to manage with your script varies greatly.

Perl continues to impress with the mind-boggling choice to be found on CPAN, but its ‘there’s more than one way to do it’ approach can leave you easily overwhelmed. Less obvious, is the magnitude of Bash extensions created to solve problems that are perhaps not best suited to any sh implementation.

We can’t help acknowledging Ruby’s power and charms.

Python has excellent library support, with rival choices considered very carefully by the community before being included in the core language. The concern to “do the right thing” is evident in every decision, yet alternate solutions remain within easy reach. At least the full adoption of the pip package manager, with Python 3.4, has ensured parity with Ruby. RubyGems provide the gem distribution format for Ruby libraries and programs, and Bundler which manages all of the gems for dependencies and correct versions. Your only problem will be finding the best guide through Ruby’s proliferation of libraries.

Read around carefully. NewLisp is not a large language, but it’s an expressive one, accomplishing much without the need of add-ons. What modules and libraries that there are address key needs, such as database and web connectivity. There’s enough to make NewLISP a useful language for the admin, but not in comparison to the other four choices.

Network security

Penetration testing and even forensic examination after an attack will fall under the remit of the hard-pressed sysadmin in smaller organisations. There are enough ready made tools available that you can roll everything you may need into a neat shell script, kept handy for different situations, but writing packet sniffers or tools for a forensic examination of your filesystem in Bash isn’t a serious option.

Perl has lost some security community mindshare since the early days of Metasploit, but the tools are still there, and are actively maintained by a large user group who aren’t about to jump ship to another language. Perl has tools like pWeb – a collection of tools for web application security and vulnerability testing – which is included in distros, such as Kali and Backbox. Tools such as Wireshark are a powerful aide to inspecting packets, but sometimes you’ll need to throw together your own packet sniffer.

NewLISP has impressive networking features, even if it lacks the pen-testing tools of the others.

Python not only has Scapy, the packet manipulation library, but provides a socket library for you to easily read and write packets directly. Ruby’s blocks (write functions on-the-fly without naming them) and other features are great for writing asynchronous network code, and its rapid prototyping matches (and even beats) Python. But Ruby’s biggest boon is Metasploit, which is the most-used pen-testing software.

In terms of ready rolled tools, you can mix and match as needed, but Perl, Python and Ruby all provide everything you need to quickly examine a network for weaknesses or compromises on-the-fly. Note: Python is featured in more security-related job adverts now.

Last, NewLISP isn’t well-known among penetration testers and grey hat hackers, but thanks to the networking built in to the language, a function call and a few arguments will create raw packets for pen testing. Once more, NewLISP has clear potential but suffers from its relatively tiny user base.

Web native scripts

Such of a sysadmin’s life has migrated to the web, so you’ll need a scripting language that has kept pace. We examined both ease of writing our own code, and finding available solutions for doing anything from web interfaces to system stats. What’s noticeable about these languages is the difference in expressiveness and style to produce similar results.

However, this is, once again, secondary to personal preference and local support for many admins. Ruby is quick and enjoyable; Python ‘feels right’ probably due to it being more human readable; newLISP is astonishingly powerful. But these observations remain partisan clichés without a supportive and maintainable environment to use and develop the code for your own networks.

  1. Bash

    While Bash will be no one’s first choice for a web programming language, it’s good to know that when your server doesn’t provide for your first choice you can fall back on it thanks to bashlib. This a shell script that makes CGI programming in the Bash shell somewhat more tolerable.
    Your script will be full of echo statements, interspersed with your commands to produce the desired output. Security considerations mean we wouldn’t recommend running this on the open Internet, but it’s worth bearing in mind that Bash works well as a prototyping language.
    It’s easy to fill a text file with comments describing the broad structure that you want, then fill in the gaps – testing snippets interactively and pasting into to check your code as you go. You’ll soon be up and running with a proof of concept.

  2. newLISP

    Code Patterns, by NewLISP creator Lutz Mueller, is available on the website and has chapters on HTTPD and CGI, as well as TCP/IP and UDP communications. If you add in the section on controlling applications, and you’ll have everything to get you started.
    NewLISP’s built-in networking, and simple (or lack of) syntax, makes it surprisingly easy to generate HTML pages of results from, for instance, your monitoring scripts. For a ready built framework, newLISP on Rockets – which uses Bootstrap, jQuery and SQLite – combines rapid application development with good performance. NewLISP on Rockets provides several functions, from (convert-json- to-list) via (twitter-search) to (display-post-box), which will help you add web functionality.
    We’re impressed but we remain concerned by the small size of the community and the intermittent pace of development.

  3. Perl 5

    Perl was the first web CGI scripting language and has more or less kept pace with the times. It certainly has the libraries, and enough examples to learn from, but with no dominant solution you’ll have to pick carefully. Catalyst, Dancer, and Mojolicious are all good web application frameworks.
    More likely you’ll find everything you need in CPAN. You can glue together a few of the libraries – many of which are already collected together in distros – to handle a pipeline of tasks, such as retrieving XML data, converting the data to PDF files and indexing it on a web page.
    Perl’s traditional CGI interface is still available, and despite better performing alternatives abstracted through PSGI, you may find that use CGI; is all you need to web-enable your script, and remember: ‘there’s always more than one way to do it’.

  4. Python

    Python’s Web Server Gateway Interface (WSGI), which was defined in PEP 333, abstracts away the web server interface, while WSGI libraries deal with session management, authentication and almost any other problem you’d wish to be tackled by middleware.
    Python also has plenty of full- stack web frameworks, such as Django, TurboGears and Pylons. Like Rails, for some purposes you may be better off coding web functionality onto an existing script. But Python’s template engines will save you from generating a mess of mixed HTML and Python.
    Python has many other advantages, from the Google App Engine cloud with its own Python interpreter, which works with any WSGI- compatible web application framework, for testing of scalable applications to supporting a clean style of metaprogramming.

  5. Ruby

    Don’t imagine for one moment that Rails is a panacea for most sysadmin problems. It’s not. And while Sinatra certainly makes it easy to roll out anything web-based in Ruby, even this is overkill for most purposes.
    That said, Rails does a good job of getting code up quickly and just doesn’t drown in all that magic, generated code. Ruby is ideal for getting any script web-enabled, thanks to gems that are written by thoughtful people who have made sane decisions.
    Putting a web interface on our backup script, for example, was fun, but distracting as we played with several gems, eg to export reports to Google spreadsheets. Tools like nanoc , which generate static HTML from HAML, and some of the reporting gems complement the language’s expressiveness, and make adding any functionality to scripts a breeze.

Scripting languages part 1

Every admin loves time-saving shortcuts, and carries a selection of scripts from job to job, as well as inheriting new ones when arriving in post. The question any new admin asks is which is the best language to learn? (Followed by, where’s the coffee?) Veterans of language wars should know that the best language question rarely has a simple or definitive answer, but we thought it would be well worth comparing the most useful choices to make your Linux life easier. Most scripting languages have been around longer than you think.

For example, NewLISP was started on a Sun-4 workstation in 1991. They’ve borrowed from each other, and elsewhere, and accumulated a long legacy of obsolete libraries and workarounds. Perl’s Regular Expressions, for instance, are now found everywhere, and in some cases better implemented elsewhere.

So what matters most? How fast the script runs, or how quickly you can write it? In most cases, the latter. Once up and running, support is needed both from libraries or modules to extend the language into all areas of your work, and from a large enough community to support the language, help it keep up with trends, and even to innovate it. So, which scripting language should you learn to improve your Linux life this year?

How we tested…

Comparisons, they say, are invidious. This is certainly true for programming languages, where personality and local support are, at least, of equal import to criteria such as speed, and the level of support for different paradigms.

Given this, we’re presenting a mixture of facts, collective opinions and our own prejudices, but it’s a basis for further investigation.

The key to a scripting language’s usefulness to the sysadmin lies not just in how easily it helps solve problems, but in how many of the solutions have already been written, and are available to download and adapt, and preferably well-documented.

We tried to work across the range of versions installed on a typical network, but insisted on Python 3. Other than that, we’ve tried to stay in the context of working with what you’re likely to find on your network.

The learning curve

The key questions are: how easy is the language to pick up? Are the learning resources at least adequate? Even if these two questions are answered in the positive, they still need to be backed up by a helpful community to assist you in quickly producing something useful, and help maintain that initial enthusiasm as you hit inevitable problems.

To produce a backup script and test scripts in each of the languages, we started by browsing Stack Overflow. But downloading random code means no consistency between Posix (pure Bourne Shell) scripts, modern Bash, and legacy code that occasionally fails.

From MOOCs to the bookshop, Python learning resources are everywhere.

Fortunately, is a great tool for checking the correctness of scripts, and teaches you best practice as it corrects them. The Linux Document Project’s (perhaps overly) comprehensive Advanced Bash Scripting Guide ( abs/html) is also excellent and will help you quickly gain confidence.

Perl’s online and built-in documentation is legendary, but we started by running through an exercise from the classic O’Reilly admin book, Running Linux , then leapfrogged the decades to No Starch’s recent Perl One- Liners by Peteris Krumins.

Those who eschew the book form should try, a source of cumulative community wisdom. Recent efforts at getting youngsters learning through Code Club (www. and the rest of us through PyConUK education sprints and open data hackdays have shown Python to be easily picked up by anyone.

But out-of-date advice, such as the many ways of running subprocesses which persist for compatibility reasons, means careful reading is needed, and it’s yet another good reason for starting with Python 3, not Python 2.

Head to www.python. org/about/gettingstarted for large list of free guides and resources. Ruby is also an easy sell to learners, and before Rails, command-line apps were what it did best.

David B. Copeland’s book, Build Awesome Command Line Applications in Ruby will save you hours of wading through online documentation, but we were able to get up and running on our test scripts with a couple of web tutorials.

Last, we come to NewLISP: a challenge to programmers schooled only in non-LISP family languages, but you’ll be amazed by what it manages to accomplish with just lists, functions and symbols. We dived right in with the code snippets page on, adapting to build our backup script, and were rewarded with terse, powerful code, that was easier to read than its equally compact Perl counterpart.

Version and compatibility

The question here is: have I got the right version? Lets start with Bash . Every modern Linux distro ships with a version that will run your scripts and anyone else’s. Bash 4 , with its associative arrays, coproc (two parallel processes communicating), and recursive matching through globbing (using ** to expand filenames) appeared six years ago.

Bash 4.2 added little, and is four years old and Bash 4.3 ‘s changes were slight. Perl is still included in the core of most distros. The latest version is 5.20 (with 5.22 soon to appear), but many stable distros ship with 5.18. No matter, you’re only missing out on tiny improvements, and just about every script you’d want to write will be fine. The switch from Python 2 to 3 still catches out the unwary.

As the Unix shell dates back decades, you will find that recent Bash versions contain a few unexpected syntax changes.

Run Python 3 if you can and check the documentation if you come unstuck. Python 3.3 is our baseline for Python 3 installs and Python 3.4 didn’t add any new syntax features.

Ruby version changes have caused enough problems that painless solutions have appeared, rvm enables you to run multiple versions of Ruby, and bundle keeps track of the gems you need for each script. NewLISP’s stability and lack of third- party scripts is an advantage here. We can’t, however, guarantee every script will run on the latest versions.