I've developed several scripts and codes for my research. Some of them might be useful to other people. These are not complete software packages, but code sniffets which can be quickly incorporated to your programs. Feel free to use and modify the following codes. However, please let me know it. (E-mail address : msshin @ kasi.re.kr)
[SDSS tools] [VO tools] [Hadoop MapReduce] [SkyDOT tools] [PyGL: Python for Gravitational Lensing] [PyCos: Python for Cosmology] [Analysis of cosmological simulations] [PyVI : visual inspection of astronomical data] [Produce multiple HTML files] [Convert FITS to ordinary graphic files] [Variability detection and period estimation] [BASH scripts to use IRAF for reducing dithered images]
I have used the following simple shell, Python, and IDL scripts to access the SDSS database. Feel free to use the scripts for your science. Because all of them are short and simple scripts, I don't provide any documents for them.
Note : the Python scripts need
the SDSS command line tool, sqlcl.
The SDSS CASJobs server also provides a web interface with SOAP. The following Python codes use this SOAP interface. These tools need WSID which is an id for CASJobs accounts (see http://casjobs.sdss.org/CasJobs/casjobscl.aspx).
The GALEX CASJobs also supports the same SOAP services which the above tools can be used with changes for different hostnames and urls. Check the GALEX webpage of the SOAP service.
- Virtual Observatory tools to access NED, Simbad, Aladin, and Vizier
usage: download_reference_from_Vizier.py [-h] [--config [CONFIG]] --output [OUTPUT] [--plot] [--catalog {gaiadr2,gaiadr2_G,psdr1_r,psdr1,smssdr1.1_r,smssdr1.1,vhsdr4}] [--min_ra [MIN_RA]] [--max_ra [MAX_RA]] [--min_dec [MIN_DEC]] [--max_dec [MAX_DEC]] [--center_ra [CENTER_RA]] [--width_ra [WIDTH_RA]] [--center_dec [CENTER_DEC]] [--width_dec [WIDTH_DEC]] [--filter] [--exbox_min_ra [EXBOX_MIN_RA]] [--exbox_max_ra [EXBOX_MAX_RA]] [--exbox_min_dec [EXBOX_MIN_DEC]] [--exbox_max_dec [EXBOX_MAX_DEC]] [--stitch STITCH] [--server {cds,cadc,cam,cfa}] [--verbose] download_reference_from_Vizier.py: error: the following arguments are required: --outputThe script already has several examples of extracting and filtering Gaia DR2, PS DR1, SMSS DR1.1, and VHS DR4 catalogs.
Documentations given in the Hadoop website are useful to help people understand basic
concepts of MapReduce and how Hadoop works. However, if you need more explanations with exmaples, I recommentd the following
books: Pro Hadoop (Jason Venner, 2009), Hadoop in Action (Chuck Lam, 2010), and Hadoop: The Definitive Guide (Tom White, 2010).
There are some papers reviews about practical issues: Data-Intensive Text Processing with MapReduce (Jimmy Lin & Chris Dyer, 2010).
Unfortunately, the most recent version of Hadoop is not fully consistent with new APIs. Therefore, when you develope your own
Java classes for data input/output or to access the Hadoop system, you still need to use old deprecated APIs. You can find how we tried Hadoop with astronomy data in
this poster.
Using the following shell scripts, you can automatically get the data file of
the
Nothern Sky Variability Survey (NSVS) from SkyDOT
site. Definitely, there are other fast ways to get data. For example, using
files after downloading the related files from the site is much faster than these
scripts. Even the implementation of the code in different languages will must be
faster than the current shell script. Yet, the shell script is quite easier than
those approaches.
- PyGL : Python for Gravitational Lensing
It's a Python + C++ package for the analysis of strong graviational lensing. The first public version will be released when the code is stable. The SWIG has been used to make Python interfaces for the C++ codes. This package also has some modules for simple cosmological calculations and observation. The following extra module is distributed independently too.
- Analysis of cosmological simulations
- PyVI : visual inspection of astronomical data
In astronomy, people somtimes check many images or spectra visually and identify
interesting objects. This Python code is for the visual inspection of image files and
leaving related logs for the images in a file. For a given list of images, it
shows images and ask people to leave any logs that can be saved in a file. The
Python Image Library
and Tkinter are
required with this code. You can check the screenshots of Ver. 1,
Ver. 2.5, and
Ver. 3. A user can put comments on images through
an interface based on Python curses module. There are multiple versions in this page.
The version 1.0 shows only images with the input list file of image filenames.
The version 2.0 shows the contents of text files corresponding the image files.
The input list file for the version 2.0 can have two columns of image filenames and
text filenames, or one column of only image filenames. The version 2.5 supports FITS files
by using the DS9 FITS viewer and
its Python XPA library (pyds9). The version 3.0
supports SAMP (Simple Application Messaging Protocol)
by using SAMPY, and supports DS9
and Aladin with their SAMP interfaces. Here, PyVI becomes a SAMP hub which communicate
with DS9 and Aladin. The version 3.0 follows the same input file formats as the version 2.5. A future version will also
support spectra stored in FITS by using VOSpec and its SAMP interface. The version 3.5 supports a general FITS viewer such as
JSky. It is also a version supporting spectrum visualization tools like
SPLAT-VO and
VOSpec with their SAMP interfaces.
This small program can be easily modified to access SQL databases and their relevant astronmoical data on either local disks or
online sites. When you encounter problems, try to use the program in the interactive
Python shell. This way will help you figure out what causes the problems.
Version 1.0
>>from PyVI import image_db >>from PyVI import curses_interface >>infn="list.txt" >>outfn="log.txt" >>dir_images="./Images/" >>db = image_db(infn, outfn, dir_images) >>db_inf = curses_interface(db, infn)
PyVI.py [list filename] [log filename] -d [the directory name of image files] Example) PyVI.py sel.list log.list -d ./Gallery -d : directory name is optional
>>from PyVI import image_db >>from PyVI import curses_interface >>infn="list.txt" >>outfn="log.txt" >>dir_images="./Images/" >>dir_texts="./Texts/" >>db = image_db(infn, outfn, dir_images, dir_texts) >>db_inf = curses_interface(db, infn)
PyVI.py [list filename] [log filename] -i [the directory name of image files] -t [the directory name of text files] Example) PyVI.py sel.list log.list -i ./Gallery -t ./Texts -i and -t : optional
>>from PyVI import image_db >>from PyVI import curses_interface >>infn="list.txt" >>outfn="log.txt" >>dir_images="./Images/" >>dir_texts="./Texts/" >>db = image_db(infn, outfn, dir_images, dir_texts) >>use_fits = 1 # if you want to use FITS images >>import ds9 >>db_inf = curses_interface(db, infn)
PyVI.py [list filename] [log filename] -f -i [the directory name of image files] -t [the directory name of text files] Example) PyVI.py sel.list log.list -f -i ./Gallery -t ./Extra_text -f : if your image files are FITS files, use this option. PyVI will use the DS9 FITS viewer. You need to open the DS9 with -xpa option. If the DS9 viewer is not available for PyVI, PyVI tries to open it. -i and -t : the directory names are optional if the directory names are not given, the current working directory is a default directory.
>>from PyVI import image_db >>from PyVI import curses_interface >>infn="list.txt" >>outfn="log.txt" >>dir_images="./Images/" >>dir_texts="./Texts/" >>db = image_db(infn, outfn, dir_images, dir_texts) >>use_fits = 1 # if you want to use FITS images with DS9. >>use_fits = 2 # if you want to use FITS images with Aladin. >>use_fits = 3 # if you want to use FITS images with both DS9 and Aladin. >>import sampy >>db_inf = curses_interface(db, infn)
PyVI.py [list filename] [log filename] --ds9 --aladin --image_dir [the directory name of image files] --text_dir [the directory name of text files] Example) PyVI.py sel.list log.list --ds9 --image_dir ./Gallery --text_dir ./Extra_text --ds9 : PyVI will use the DS9 as an image viewer which supports FITS files. You must check the DS9 first whether it is available with the SAMP. --aladin : PyVI will use the Aladin as an image viewer which supports FITS and other file formats such as JPEG and PNG. You must check the Aladin first whether it is available with the SAMP. If both --ds9 and --aladin options are selected, both viewers are used in the PyVI. --image_dir and --text_dir : the directory names are optional if the directory names are not given, the current working directory is a default directory.
>>from PyVI import image_db >>from PyVI import curses_interface >>infn="list.txt" >>outfn="log.txt" >>dir_images="./Images/" >>dir_texts="./Texts/" >>db = image_db(infn, outfn, dir_images, dir_texts) >>use_fits = 1 # if you want to use FITS images with DS9. >>use_fits = 2 # if you want to use FITS images with Aladin. >>use_fits = 3 # if you want to use FITS images with both DS9 and Aladin. >>import sampy >>db_inf = curses_interface(db, infn)
PyVI.py [list filename] [log filename] --ds9 --aladin --image_dir [the directory name of image files] --text_dir [the directory name of text files] Example) PyVI.py sel.list log.list --ds9 --image_dir ./Gallery --text_dir ./Extra_text --ds9 : PyVI will use the DS9 as an image viewer which supports FITS files. You must check the DS9 first whether it is available with the SAMP. --aladin : PyVI will use the Aladin as an image viewer which supports FITS and other file formats such as JPEG and PNG. You must check the Aladin first whether it is available with the SAMP. If both --ds9 and --aladin options are selected, both viewers are used in the PyVI. --general : PyVI will use a general FITS image viewer including JSky, DS9, and Aladin. You must check the viewer first whether it is available with the SAMP. However, do not use --general with either --ds9 or --aladin together. This option is specially for FITS viwer showing only a single frame. Therefore, this option is appropriate for tools like JSky. --spec : PyVI will use a spectrum viewer (e.g. SPLAT-VO, VOSpec). You must check the viewer first whether it is available with the SAMP. --image_dir, --text_dir, --spec_dir : the directory names are optional if the directory names are not given, the current working directory is a default directory.
- Python code to produce multiple HTML files with objects list
Check the code. You can use this simple script to produce multiple HTML files which show the list of objects with images and links to external web sites. The code also has simple functions to convert angles given in degree to sexagesimal formats. If you want to get only these functions, check coordinates_conversion.py below.
- Python code to convert FITS to ordinary graphic files
Please, check this link. If you need a simple way, please use fits2jpeg.sh given below.
- Variability detection and period estimation
Please, check this link.
- Bash shell scripts to use IRAF for reducing dithered images (of NIR)
You can find details about this package of shell scripts in this link.
- Miscellaneous codes
Useful list or table
--------------------------------
"Controlling complexity is the essence of software development." - Brian Kernigan
--------------------------------