Saving ACP image data to a cloud drive

[John Pye]

In our campus observatory we have 3 telescopes running The SkyX Pro, MaxIm 5.23, ACP 6 (forget exactly which 6 version it is - sorry), and FocusMax. We're controlling everything at each station with a Windows 7 Pro laptop and all works fine, including the web interface and automated imaging over the web using scripts. We mainly intended the laptops to be control computers only and not storage computers (purchased a while ago with limited budget), so they only have 500 GB hard drives. We also have 2 TB cloud drives in the observatory, one dedicated to each scope to store and backup the data. I am able to open Windows Explorer and see the networked drives and can drag data files from any of the laptops to any of the cloud drives through Explorer. What we'd like to do is be able to save the data files directly to the cloud drives (especially when doing automated imaging runs). I've read the documentation about setting up imaging paths to folders on the local machine and also Bob's info on Dropbox, but not sure if Dropbox is the right answer for us. When I go into ACP Preferences to try to set the image path to the cloud drives, they don't show up like another hard drive or location on the local machine would, even though I can see them in the Network folder in Windows Explorer. Besides saving space on the small control laptops, this functionality would allow us to drag these files to other networked machines we have that connect to the cloud drives dedicated to image processing that our students can work on. But at the moment I have to manually drag the files from the control laptops to the cloud drives in Explorer after the imaging runs are done and then unzip them so the students can access them. I'd really like to be able to do this in closer to a real-time environment if it's possible so the images are downloaded to the cloud drives as they're collected and students can work with them while the imaging sequence continues.

Is there a way to save data from from ACP to a cloud drive like this rather than on the local machine as is the default? Is it a difference between how Windows talks to and sees those network drives and how ACP does (or doesn't)? I'm no expert on this and was hoping ACP would just recognize the cloud drives like another hard drive or directory path but it doesn't using the methods I've tried so far.

Any suggestions on how I might be able to set this up? Is it possible? Thank you for any help/suggestions you can provide.

JP

[Robert Capon]

Hi John:

There are a lot of ways that this can be done. You can change the path of where you save the files by going to ACP Prefs, Local User, click on "OK, then how do I have ACP put image files elsewhere", and click on the "Where are my files tool".

But here's the issue. If you don't save the files on the local computer you might slow down MaximDL's file-saving efficiency while you're in the middle of an automated run because the files can be very large. (My files are 21 MB and the calibrated files are 42 MB, so that's 63 MB per image and I'm doing over 120 per night). I think that could overwhelm DropBox and might slow down Maxim if you're saving massive files over a local network.

One solution that's worked well for me is a program called "watchDirectory". This is a really powerful program that you can set up to watch a particular directory and set up macros to do lots of things including compressing files by zipping them, copying them to another directory (including shared folders over a network), deleting them after they've been copied, sending files by FTP, etc. Each of these actions is called a "Task", and the program is infintely flexible. For example, you could decide to copy all of the files except ignore files with the word "RAW" in the name, so that only your calibrated images would be copied.

The advantage of something like this is that Maxim can save files on the local machine without putting file transport drag/overhead while saving files during automated observing. But once Maxim saves the file, watchDirectory would monitor the files in real-time and transport in the background while ACP and Maxim continue to do their thing.

I'm sure there are lots of ways to do what you're trying to do, so a program like watchDirectory is just one solution. But I can confirm that I'm using it, and it plays very nicely with ACP and Maxim.

Best,

Rob

[John Pye]

Thank you for this information Rob. This is very useful! I have browsed around the website for Watch Directory and it definitely looks like it could do what I need. And after reading your post about the file transport drag/overhead issues that could result during automated imaging, it seems like this is a much better way to do my image file moves to a network storage drive. I will do some more research on this and most likely give this a trial run. It may be the solution I'm looking for.

Thank you again for your response and the great information!

JP

[Robert Capon]

Hi John-

You're most welcome. I think they offer a free trial period on the software.

Other folks might chime in with other solutions, but this one works for me.

Good luck!

Rob

[Jeff Lewis]

Google Drive is a very efficient way to do this. I have been using it for a couple of years now. Once set up on your system, you define sync folders. When files are added to these folder hey are synced to Google Drive. The files are stored locally, so you aren't incurring any network overhead in the writing of the file. If you are offline, files will be synced the next time you are online. You can configure bandwidth utilization to keep that within limits I don't believe any cloud storage actually has you writing across the network as the file is saved, even DropBox. This would be disastrous if you where off the network. (I avoid Dropbox due to their horrendous security and their terms of usage) . Anyway I have been using Google dive and Open Drive with ACP for a couple of years with zero problems. The simplest solution is always better.

[Robert Capon]

Jeff-

That sounds like a neat solution.

Best,

R

[John Pye]

Thank you Jeff. I appreciate your input. I now have several options to consider to see which will work best for our situation.

[Bob Denny]

For reference this is covered in ACP Help:

Snap1.png

Snap2.png

[John Pye]

Hi Bob. Thanks for the reference. I have read through this but had pretty much decided Dropbox might not work for us because of the sync features (where the files might get changed due to processing and then be changed in the folders with the raw data). I know we'd have to create copies as working files (which is easy enough) to avoid this, and there are ways to work with this (even with students accessing data files to work on). I was just interested to see what other folks were doing and have gotten some good ideas for consideration. I'm not sure if GoogleDrive and BitTorrent have the same potential issues as Dropbox, and I was also looking at other ways of moving data around and syncing our network. I will now pursue these options further. No need for you to reply to this - just wanted to thank you again for always providing great service to ACP customers.

[Bob Denny]

Actually, ACP is Drop-Box friendly... it does all of its processing on temporary files, then moves only the final product into the final location. However once the files have been sync-delivered to the students, they should copy them to local non-synced locations (or other synced folders) in order to prevent them from altering the originals on the observatory (as you note). Or you could ask them to move the files to their own local storage, and that will remove them from the observatory, freeing up space there. This works best if each student has their own ACP account, preventing them from messing with other students' data!!

With the above said, I noticed that you aren't a pre-sale/eval person, instead it looks like you have a personal internet license that expired in 2008. If this is so, then your very old version of ACP does not have the DropBox friendly features, nor many others :-) Are you actually a school? And are you running from a different license? I looked because you were using this free/pre-sales section of the Comm Center. If you aren't the same person or you are using a different license, then my apologies...