Interact with the web, unix style.
Pigshell is inspired by Unix philosophy: Everything is a file. Programs should do one thing well. Create tools by stringing together a combination of simpler tools. Like human language, CLIs give us freedom of expression - the ability to generate, using a limited vocabulary, an immense variety of meaningful sentences needed to deal with the immense variety of complex situations thrown up by the environment. In contrast, authoritarian UIs restrict us to a menu of pre-approved cliches, with little scope for novel usage and adaptation beyond the designer's imagination.
Running the psty server on your desktop is strongly recommended.
It exposes a local directory for pigshell to use as a
/home, serves as a
proxy HTTP server, and lets pigshell pipe web data through desktop unix
Pigshell is under active development. No commands, APIs or interfaces are frozen at this point. We expect to release the sources under the GNU GPLv3 around June 2014. A few basic examples of pigshell usage are given below.
Click on the first example in the sidebar, or type the following at the prompt:
cat http://pigshell.com/sample/life-expectancy.html | table2js -e "table.wikitable tr" foo country data | template -ig /templates/d3-worldmap1
A pigshell pipeline processes streams of objects. Commands should be
considered as generator functions yielding objects, composed using the pipe
operator. The pipeline starts when the last member (an implicit
the upstream command for an object, which in turn asks its upstream command and
so on, until the command at the head reluctantly yields an object. It is
processed and "returned" downstream, until it hits
Stdout which displays it
on the terminal.
Stdout has an insatiable appetite for objects, so it asks
for one more, and the process continues until a
null object, signifying the
end of the pipeline, makes its way down. Unlike Unix commands, pigshell
commands are not independently executing processes.
Using the Terminal
The terminal should be familiar to Unix/bash users, featuring tab completion, history, and Emacs-style CLI editing shortcuts.
The primary prompt is
pig<basename_of_cwd>$. When a command is running,
a secondary prompt of
> is displayed, which can be used to typeahead
Ctrl-C and Ctrl-Z kill and pause the current foreground command
respectively, while Ctrl-B "backgrounds" it (roughly like an Ctrl-Z +
bg in bash).
A command sequence (e.g.
ls|sum; sleep 10; echo done) issued on the CLI will
fill its output area as and when objects are emitted. The prompt associated
with a command sequence glows green or amber while it is running or stopped,
and on its completion settles to red or black depending on its exit status.
cat work similarly enough to their Unix namesakes to
help you explore the system.
ps command displays a list of running pipelines. To kill a long-running
ps to find its PID and
kill. You can also
Attaching Data Sources
To attach Facebook, Google Drive, Picasa, click on the corresponding
icon under Data Source in the sidebar. Attaching an account automatically
mounts associated filesystems under
mount command without arguments displays the list of currently
Everything is a file. Friends are files too. After attaching your Facebook account,
cd /facebook/friends; ls
will give you a list of your friends.
Where in the world are my friends?
map is a command which plots files with location attributes on a map.
Another way of doing this would be
ls /facebook/friends/ | map
Pigshell passes objects over pipes. In this case,
ls emits a stream of
file objects, which are consumed by map.
Let's refine the above query: Where are all my male friends?
ls /facebook/friends | grep -f gender "^male" | map
grep is a generic filter command, which may filter either by an object's text representation, or a specific field - in this case, gender.
How many friends do I have?
ls /facebook/friends | sum
Pie chart of relationship status of all female friends
ls /facebook/friends | grep -f gender "female" | chart -f relationship_status
$HOME sweet $HOME
/home. Download Psty, run it on your desktop:
python psty.py -a -d /some/dir # Run in DESKTOP SHELL (bash), not pigshell
and on pigshell,
mount http://localhost:50937/ /home # Run in PIGSHELL, not desktop
The psty server runs only on Linux and Mac OS at present.
Now you can read and write to
/home it will be backed by
cat images and PDFs stored on your desktop inside
This mount command needs to be typed every time you start or reload the page. To do it automatically,
echo "HOME=/home; mount http://localhost:50937/ $HOME" >/local/rc.sh
/local/rc.sh is a script stored in the browser's LocalStorage and will be invoked every time http://pigshell.com is (re)loaded. You need to create a /local/rc.sh on every browser on which you use pigshell.
Assuming you're running psty, backing up Google Drive to your desktop is as simple as
mkdir /home/drivebackup cp -rv -X /Trash /firstname.lastname@example.org /home/drivebackup
More details on using Google Drive with pigshell.
Backing up a Picasa album:
mkdir /home/foo; cp /picasa/foo/* /home/foo
Similarly, creating an album and uploading a bunch of pictures to Picasa:
mkdir /picasa/bar; cp /home/barpics/*JPG /picasa/bar
(note that album creation and uploads to Picasa require psty's proxy services)
Copying random URLs to your desktop also works:
cp -c http://ftp.freebsd.org/pub/FreeBSD/ISO-IMAGES-amd64/10.0/FreeBSD-10.0-RELEASE-amd64-bootonly.iso /home
-c option continues where it left off, so you can resume interrupted
If you don't have psty:
You can copy the files to
/downloads, and it will get to your browser's default download folder. Note that you cannot see anything inside the
/downloadsdirectory, it's just a pseudo-target to trigger a browser download. For example,
cp /picasa/foo/DSC_1290.JPG /downloads
Click on Upload Files and select a file or files from your desktop. These files are now visible under the directory
lsto verify that they're there. Now use
cpto copy them to the target directory.
cp /uploads/cat.jpg /gdrive cp /uploads/cat.jpg /facebook/me/albums/MyCat/
read() operations generate blobs. It is up to the consuming
command to convert incoming data into the type it likes. In the command
cat returns a blob. The terminal figures out that the blob contains PNG data,
and displays it as a canvas. Similarly,
is detected as a PDF and displayed using pdf.js. In case it could not figure out the contents, it attempts to convert it to text and displays it as the usual control-character porridge (though it is mercifully silent, unlike Unix terminals)
In some cases, you have to manually convert data between stages in the pipeline. For example,
cat http://pigshell.com/sample/README.md | to text | jf 'x.split("\\n")' | sum
implements a poor man's
cat returns a blob,
to converts it to text,
jf splits it into lines,
sum counts the number of objects it gets. A
... | sum would have returned 1, since only one object (a blob) was presented
URLs as files
Absolute URLs can be used in most places where a file path is expected. Mounting an HTTP URL exposes all links within that page as directories. To mount arbitrary, non-CORS-enabled URLs, you need to run psty.
mount http://pigshell.com/sample/ /mnt; cd /mnt; ls cat oslogos.png cat . cat . | to text
Processing on the desktop - Wsh
Psty runs a websocket service, effectively converting every Unix utility which uses stdin/stdout into a potential member of the pigshell pipeline. For instance, if you have ImageMagick installed,
cat http://pigshell.com/sample/oslogos.png | wsh /usr/local/bin/convert -implode 1 - - | to -g blob
will grab a png file from the web, pipe it through ImageMagick on the desktop, and display the result in pigshell.
To visualize disk usage in a zoomable treemap,
wsh du /Users/foo | to -g text | template -ig /templates/d3-du-treemap
du of a deep tree may take a while, try with a shallow directory
Finally, an example from 7 command-line tools for data science. This assumes that you have R and ggplot2 already installed.
cd /home; cp https://raw.github.com/jeroenjanssens/data-science-toolbox/master/tools/Rio .
Rio is a bash script to coax R into running with stdin/stdout.
cat https://raw.github.com/pydata/pandas/master/pandas/tests/data/iris.csv | wsh bash ./Rio -ge 'g+geom_point(aes(x=SepalLength,y=SepalWidth,colour=Name))' | to -g blob
Frequently Asked Questions
What about the privacy of my data?
What browsers are supported?
Pigshell should work on most modern browsers. We use Chrome on MacOS as our primary dev/test platform, but Firefox, Safari on MacOS, Chrome and Firefox on Linux, and Chrome on Windows work as well.
Firefox on Windows and Linux occasionally runs into stack overflows. There is no plan to support IE.
Nothing on the iPad works currently due to keyboard input issues.
ls | catin pigshell behave like
ls | xargs catin Unix?
Pigshell passes objects (rather than opaque data) over the pipe.
lsin its normal incarnation emits file objects.
cat(and other filter commands) receiving file objects will process them not as text, but as files. You can think of it as an implicit
ls -lemits text strings, so
ls -l | catwill behave the same on both pigshell and a Unix shell. While initially confusing, this is the more natural behaviour in a web environment thickly populated with structured objects. See the User guide for more details.
I can't see the cursor.
Long-running or CPU-intensive commands may sometimes freeze the page for a few seconds. It is also possible that keyboard focus has gone elsewhere. Click on the last prompt and focus should return there.
There are two cursors on the screen. What do I do now?
You probably ran a command which is reading something from standard input, and it has opened up a little line just below its command line with a cursor. You may click on that line to move focus there and enter input. You can use Ctrl-D at the beginning of a new line to signal end of input.
If it was a mistake, then you can simply click on the latest prompt and continue issuing new commands. You can ignore or kill the old command.
How do I reboot?
Reload the page. Hold down Shift while reloading may help, clearing your browser cache.
What commands are available?
Most commands are built-in, while a few are scripts found in
/bin. You can see a list of commands using the
helpcommand. Help for a specific command like
lsmay be seen either using
How do I write my own scripts?
The syntax is close enough to bash, and looking through existing scripts in
/binshould give you enough of an idea to start writing your own. A trip to the User Guide is definitely recommended, though.
editcommand implements a simple, minimal editor using CodeMirror.
You could also create/edit scripts on your desktop using your favourite editor and run them from somewhere under
How do I store scripts so that they survive a "reboot"?
The /local filesystem uses the browser's HTML5 localStorage as its backing store. Files stored there will survive a page reload (aka "reboot") Note that localStorage is typically limited to about 5 MB per site, so this is only suitable for saving small files like scripts.
Much better to keep it inside /home, as this area is backed by your host filesystem.
How do I figure out what attributes a file object contains?
printf -j <file>. Most files have a
rawattribute containing all the information returned by the backend API.
How do I make sense of these errors? I get parse error for a line which is most certainly correct. Expected "#", "\n", "\r", "\r\n"? WTF?
Sorry. Error reporting is still in the "PC Load Letter" era. The line number indicates the beginning of a block which failed to parse. So if you have a long multi-line
ifconstruct with an error somewhere in the middle, it will flag the
ifline as the source of the error.
The best way right now is to comment out chunks of the block to figure out which one is causing the real trouble.
What data sources are supported, and what operations work on those files?
Facebook: Creating new albums, reading (but not editing!) photos, writing new photos. Photos created with Pigshell can be deleted. Reading of posts, writing of text posts.
Google Drive: Reading, creating, deleting files.
Picasa: Reading and editing of photos. Creating and deleting photos supported if you are running psty, as they need a proxy to duck CORS issues.
The user guide has more detailed coverage of pigshell concepts and the scripting language.