There is a golden rule by which I live when it comes to programming, if I have to do it more than once it gets automated, scripted up and generally I write a short post to remind myself how and why I did it.
Throughout the course of my work I regularly need to access remote servers to archive and convert data for analysis. Due to permission restrictions, this often leads to creative approaches with data moving back and forth across networks to a machine that has the grunt to do the processing, then over to another machine with ample storage to maintain a decent time series. This is generally achieved through the use of shell scripts, specialist languages (like Python or NCL) and common libraries like NCO or CDO, all of which are scheduled and executed by cron. This usually works a dream.
So imagine my surprise when my magical setup was for all intents and purposes "working" but I was getting no output from my code?
I checked the cron logs, nothing there. I checked that the data was moving, looks ok. I checked permissions on the source and target directories across all of the machines, all good. So what was the problem?
It didn't come to me until I was editing my bash profile, where I have a myriad of magical aliases and path appendices that I wondered if cron actually used all of these things? Checking my scripts, I make use of these shortcuts on a regular basis so I decided to run a test. I popped the following into a crontab and waited for the result.
* * * * * env > /tmp/env.log
Reading this log file it became immediately obvious what my problem was.
Cron does not obtain the full user environment
So here I was spending all of this time trying to figure out why things weren't happening via cron when I could make them happen interactively and it turns out that all I needed to do was duplicate a few of my magical aliases from my bash profile and copy them into the scripts themselves.
The take away lesson here is that automation and shortcuts can only get you so far, sometimes you have to commit a little code duplication faux pas and hard code things into a script called via cron.
I hope that this helps someone else out there.
One final thought: Although I haven't tried it yet, it is probably just as easy to source your bash profile at the head of your cron scripts to get access to all of your path aliases etc. as good practice. I'm not sure if this will work and/or will have unintended consequences, but it might be worth a shot.