Min-Su's SDSS tools and other codes
Loading

I've developed several scripts and codes for my research. Some of them might be useful to other people. These are not complete software packages, but code sniffets which can be quickly incorporated to your programs. Feel free to use and modify the following codes. However, please let me know it. (E-mail address : msshin @ kasi.re.kr)

[SDSS tools] [VO tools] [Hadoop MapReduce] [SkyDOT tools] [PyGL: Python for Gravitational Lensing] [PyCos: Python for Cosmology] [Analysis of cosmological simulations] [PyVI : visual inspection of astronomical data] [Produce multiple HTML files] [Convert FITS to ordinary graphic files] [Variability detection and period estimation] [BASH scripts to use IRAF for reducing dithered images]

- SDSS tools

I have used the following simple shell, Python, and IDL scripts to access the SDSS database. Feel free to use the scripts for your science. Because all of them are short and simple scripts, I don't provide any documents for them.

Note : the Python scripts need the SDSS command line tool, sqlcl.

  • sdss_cutout_1min.sh : it downloads the SDSS cutout image of 1 minute for a given RA and DEC.
  • sdss_mosaic_down.sh : it extracts the SDSS mosaic image from the IRSA web service for a given RA/DEC or the name of the object. (You can get a Python code for the same purpose. The code, SDSS_cutout_IRSA.py, reads a coordinate file and download FITS and JPEG files from the IRSA finder chart service. Please, check the source code and explanations given in the code.) If you want to combine the SDSS images from the SkyServer, check Stefano Zibetti's web page.
  • sdss_image_check.sh : it checkes whether a given position is included in the SDSS imaging field by sending a query to the SDSS FOOT server.
  • sdss_image_download_and_plot.tar : it is a simple shell script to download a SDSS image and then to plot a slit on the image by using the DS9. You need to know the run, rerun, camcol, field, ra, and dec of the SDSS object which you want to put a slit on.
  • sdss_photo_check.py : it searches the SDSS photometric objects for a given RA/DEC in the SDSS PhotoObjAll catalog.
  • sdss_spec_check.py : it is similar to sdss_photo_check.py. However, it searches the SDSS SpecObjAll catalog.
  • sdss_photo_flag.py : the SDSS photo flag in hexcode is converted to string. You need PhotoFlags.dat file too.
  • sdss_prim_target_flag.py : the SDSS primary target flag in hexcode is converted to string. You need PrimTargetFlags.dat file too.
  • run_composite.pro : this IDL routine produces a composite spectrum for a given list of SDSS spectra. If you want to use it for spectra from different observations, the routine simply needs to be modified for only the part of reading spectra. The current code considers the Galactic extinction, but it doesn't have the effect of velocity dispersion.
  • The SDSS CASJobs server also provides a web interface with SOAP. The following Python codes use this SOAP interface. These tools need WSID which is an id for CASJobs accounts (see http://casjobs.sdss.org/CasJobs/casjobscl.aspx).

  • SDSS_SOAP_UploadData.py : it uploads a file into a new table on the SDSS CASJobs' MyDB. Two command-line options of the table name and the input file are required. You need to change WSID and its password in the code before you use. The input file must be in a CSV format with the first row of column names. If you find that the above version does not work, try this version SDSS_SOAP_upload_DR9.py which works with the SDSS DR9.
  • SDSS_SOAP_ExecuteQuickJob.py : it runs a quick job on the SDSS CASJobs server. The example SQL query in the file creates a new table on MyDB.
  • SDSS_SOAP_SubmitJob.py : it submits a long job to the SDSS CASJobs server. The return value is the job id on the server. The example SQL query is same as the above code. HTTP POST method is used in this code.
  • SDSS_SOAP_SubmitExtractJob.py : it submits a job for output. The job id is returned.
  • SDSS_SOAP_GetJobStatus.py : it checked a status of a job on the SDSS CASJobs.
  • The GALEX CASJobs also supports the same SOAP services which the above tools can be used with changes for different hostnames and urls. Check the GALEX webpage of the SOAP service.

    - Virtual Observatory tools to access NED, Simbad, Aladin, and Vizier

  • download_reference_from_Vizier.py : a simple Python script that can be used to download a catalog from Vizier for a requested RA, DEC range. The usage of the script is
        usage: download_reference_from_Vizier.py [-h] [--config [CONFIG]] --output
                                                 [OUTPUT] [--plot]
                                                 [--catalog {gaiadr2,gaiadr2_G,psdr1_r,psdr1,smssdr1.1_r,smssdr1.1,vhsdr4}]
                                                 [--min_ra [MIN_RA]]
                                                 [--max_ra [MAX_RA]]
                                                 [--min_dec [MIN_DEC]]
                                                 [--max_dec [MAX_DEC]]
                                                 [--center_ra [CENTER_RA]]
                                                 [--width_ra [WIDTH_RA]]
                                                 [--center_dec [CENTER_DEC]]
                                                 [--width_dec [WIDTH_DEC]] [--filter]
                                                 [--exbox_min_ra [EXBOX_MIN_RA]]
                                                 [--exbox_max_ra [EXBOX_MAX_RA]]
                                                 [--exbox_min_dec [EXBOX_MIN_DEC]]
                                                 [--exbox_max_dec [EXBOX_MAX_DEC]]
                                                 [--stitch STITCH]
                                                 [--server {cds,cadc,cam,cfa}]
                                                 [--verbose]
         download_reference_from_Vizier.py: error: the following arguments are required: --output
      
    The script already has several examples of extracting and filtering Gaia DR2, PS DR1, SMSS DR1.1, and VHS DR4 catalogs.
  • NED_batch_compact_parser.py : a short Python script to parse result files from NED batch job with "compact" output type. In particular, the current code is optimized to my favorite style of NED batch job: example. Please, visit the NED site for references about batch jobs. It's quite easy to understand the code. So, please, check the source code to figure out how you can use it and how the script works. (I know that some VO tools provide an access to NED. However, I believe that batch jobs are faster than VO access when you have many objects to be searched.) For "standard" output, you can use NED_batch_standard_parser.py that prints out name, type, distance, and redshift of found objects. It's easy to understand the code.
  • Simple tools for Simbad batch jobs : for a given file that has coordinates of objects, simbad_query_builder.py generates a batch query for Simbad. simbad_script_submit.sh submits the batch query to Simbad and receives the returned results into a file. simbad_query_postprocess.py postprocess the result file. You can find how to use the tools in README file. The Simbad site explains how to use a script. You can get my codes in this link.
  • Example of Bash shell scripts to use Aladin : this file shows how you can used Aladain in a script mode by writing simple shell scripts.
  • Vizier on CDS also has provides a batch job. My python scripts help producing query files for Vizier. You can find how to use the tools in this link.
  • - Using Hadoop MapReduce

    Documentations given in the Hadoop website are useful to help people understand basic concepts of MapReduce and how Hadoop works. However, if you need more explanations with exmaples, I recommentd the following books: Pro Hadoop (Jason Venner, 2009), Hadoop in Action (Chuck Lam, 2010), and Hadoop: The Definitive Guide (Tom White, 2010). There are some papers reviews about practical issues: Data-Intensive Text Processing with MapReduce (Jimmy Lin & Chris Dyer, 2010).
    Unfortunately, the most recent version of Hadoop is not fully consistent with new APIs. Therefore, when you develope your own Java classes for data input/output or to access the Hadoop system, you still need to use old deprecated APIs. You can find how we tried Hadoop with astronomy data in this poster.

    - SkyDOT tools

    Using the following shell scripts, you can automatically get the data file of the Nothern Sky Variability Survey (NSVS) from SkyDOT site. Definitely, there are other fast ways to get data. For example, using files after downloading the related files from the site is much faster than these scripts. Even the implementation of the code in different languages will must be faster than the current shell script. Yet, the shell script is quite easier than those approaches.

  • get_light_curve.sh : it downloads light curve data of only good data points for object ids that are given in the file count.list. The script uses wget command to get the files from SkyDOT site. In order to understand the script, you must reference the NSVS paper.
  • get_objid_coord.sh : this script is similar to the above script. However, it extracts the list of object ids for a given RA and DEC with an 120 arcmin search radius. You have to know that the maxmimum number of objects are limited by the SkyDOT server. The example of usage is ./get_objid_coord.sh 10:00:00 -5:00:00 set1, where set1.list file has the list. The object ids from this code can be used in the above code.
  • - PyGL : Python for Gravitational Lensing

    It's a Python + C++ package for the analysis of strong graviational lensing. The first public version will be released when the code is stable. The SWIG has been used to make Python interfaces for the C++ codes. This package also has some modules for simple cosmological calculations and observation. The following extra module is distributed independently too.

  • PyCos(Python for Cosmology) : PyCos-1.2.tar.gz (source code distribution) You can easily install the module. It uses GNU Scientific Library (GSL) for numerically intensive calculation. Because the internal code was written in C++, you can use it with a C++ library code. By using SWIG, I also produce Python interfaces with the C++ codes. Before you install this moudle, you need to install the GSL. The SWIG is generally included in most Linux distributions. The installation and usage of this module is simple; README. The future version of this module will have more implementations. Please, check the document generated with DoxyGen for further usage. In short, the module can be used to calculate various kinds of distances, volumes, and age for given cosmological parameters. You can also get numbers about Schechter function quickly. It's a free software. Yet, I'll be glad to hear any suggestions, applications, and bugs to improve the current version.
  • - Analysis of cosmological simulations

  • GalDen : a parallel C code that measures a local number and mass density around galaxies in a simulation box. This is a part of a grouping code for the analysis of simulation data. It requires MPI and Gloabl Arrays libraries. Instead of optimizing MPI communication, I concentrated on the usage of large distributed memory. It's a free software. However, please, let me know when you use the code. I'm interested in the further optimization and improving the code for various purposes. The current version works well with about 10^7 galaxies.
  • - PyVI : visual inspection of astronomical data

    In astronomy, people somtimes check many images or spectra visually and identify interesting objects. This Python code is for the visual inspection of image files and leaving related logs for the images in a file. For a given list of images, it shows images and ask people to leave any logs that can be saved in a file. The Python Image Library and Tkinter are required with this code. You can check the screenshots of Ver. 1, Ver. 2.5, and Ver. 3. A user can put comments on images through an interface based on Python curses module. There are multiple versions in this page. The version 1.0 shows only images with the input list file of image filenames. The version 2.0 shows the contents of text files corresponding the image files. The input list file for the version 2.0 can have two columns of image filenames and text filenames, or one column of only image filenames. The version 2.5 supports FITS files by using the DS9 FITS viewer and its Python XPA library (pyds9). The version 3.0 supports SAMP (Simple Application Messaging Protocol) by using SAMPY, and supports DS9 and Aladin with their SAMP interfaces. Here, PyVI becomes a SAMP hub which communicate with DS9 and Aladin. The version 3.0 follows the same input file formats as the version 2.5. A future version will also support spectra stored in FITS by using VOSpec and its SAMP interface. The version 3.5 supports a general FITS viewer such as JSky. It is also a version supporting spectrum visualization tools like SPLAT-VO and VOSpec with their SAMP interfaces. This small program can be easily modified to access SQL databases and their relevant astronmoical data on either local disks or online sites. When you encounter problems, try to use the program in the interactive Python shell. This way will help you figure out what causes the problems.
    Version 1.0

  • Download : PyVI_ver1.py (screen shot)
  • Usage I :
    >>from PyVI import image_db
    >>from PyVI import curses_interface
    >>infn="list.txt"
    >>outfn="log.txt"
    >>dir_images="./Images/"
    >>db = image_db(infn, outfn, dir_images)
    >>db_inf = curses_interface(db, infn)
    
  • Usage II :
    PyVI.py [list filename] [log filename] -d [the directory name of image files]
                    Example) PyVI.py sel.list log.list -d ./Gallery
                    -d : directory name is optional
    
  • A list file is a simple blank or tab-separated file that has image filenames on the first column. A log file will have your comments on images as an extra column to the list file. If image files are under a specfic directory, you can set the directory with -d option.

  • Version 2.0
  • Download : PyVI_ver2.py
  • Usage I :
    >>from PyVI import image_db
    >>from PyVI import curses_interface
    >>infn="list.txt"
    >>outfn="log.txt"
    >>dir_images="./Images/"
    >>dir_texts="./Texts/"
    >>db = image_db(infn, outfn, dir_images, dir_texts)
    >>db_inf = curses_interface(db, infn)
    
  • Usage II :
    PyVI.py [list filename] [log filename] -i [the directory name of image files] -t [the directory name of text files]
                    Example) PyVI.py sel.list log.list -i ./Gallery -t ./Texts
                    -i and -t : optional
    
  • A list file is a simple blank or tab-separated file that has image filenames on the first column and text filenames on the second column. A log file will have your comments on images as an extra column to the list file.

  • Version 2.5
  • Download : PyVI_ver2p5.py (screen shot)
  • Usage I :
    >>from PyVI import image_db
    >>from PyVI import curses_interface
    >>infn="list.txt"
    >>outfn="log.txt"
    >>dir_images="./Images/"
    >>dir_texts="./Texts/"
    >>db = image_db(infn, outfn, dir_images, dir_texts)
    >>use_fits = 1 # if you want to use FITS images
    >>import ds9
    >>db_inf = curses_interface(db, infn)
    
  • Usage II :
    PyVI.py [list filename] [log filename] -f -i [the directory name of image files] -t [the directory name of text files]
                    Example) PyVI.py sel.list log.list -f -i ./Gallery -t ./Extra_text
                    -f : if your image files are FITS files, use this option. PyVI will use the DS9 FITS viewer. You need to open the DS9 with -xpa option. 
                         If the DS9 viewer is not available for PyVI, PyVI tries to open it.
                    -i and -t : the directory names are optional
                    if the directory names are not given, the current working directory is a default directory.
    

  • Version 3.0
  • Download : PyVI_ver3.py
    Screenshot :
  • Usage I :
    >>from PyVI import image_db
    >>from PyVI import curses_interface
    >>infn="list.txt"
    >>outfn="log.txt"
    >>dir_images="./Images/"
    >>dir_texts="./Texts/"
    >>db = image_db(infn, outfn, dir_images, dir_texts)
    >>use_fits = 1 # if you want to use FITS images with DS9.
    >>use_fits = 2 # if you want to use FITS images with Aladin.
    >>use_fits = 3 # if you want to use FITS images with both DS9 and Aladin.
    >>import sampy
    >>db_inf = curses_interface(db, infn)
    
  • Usage II :
    PyVI.py [list filename] [log filename] --ds9 --aladin --image_dir [the directory name of image files] --text_dir [the directory name of text files]
    		Example) PyVI.py sel.list log.list --ds9 --image_dir ./Gallery --text_dir ./Extra_text
    		--ds9 : PyVI will use the DS9 as an image viewer which supports FITS files. You must check the DS9 first whether it is available with the SAMP.
    		--aladin : PyVI will use the Aladin as an image viewer which supports FITS and other file formats such as JPEG and PNG. You must check the Aladin first whether it is available with the SAMP. If both --ds9 and --aladin options are selected, both viewers are used in the PyVI.
    		--image_dir and --text_dir : the directory names are optional
    		if the directory names are not given, the current working directory is a default directory.
    

  • Version 3.5
  • Download : PyVI.py
  • Usage I :
    >>from PyVI import image_db
    >>from PyVI import curses_interface
    >>infn="list.txt"
    >>outfn="log.txt"
    >>dir_images="./Images/"
    >>dir_texts="./Texts/"
    >>db = image_db(infn, outfn, dir_images, dir_texts)
    >>use_fits = 1 # if you want to use FITS images with DS9.
    >>use_fits = 2 # if you want to use FITS images with Aladin.
    >>use_fits = 3 # if you want to use FITS images with both DS9 and Aladin.
    >>import sampy
    >>db_inf = curses_interface(db, infn)
    
  • Usage II :
    PyVI.py [list filename] [log filename] --ds9 --aladin --image_dir [the directory name of image files] --text_dir [the directory name of text files]
    		Example) PyVI.py sel.list log.list --ds9 --image_dir ./Gallery --text_dir ./Extra_text
    		--ds9 : PyVI will use the DS9 as an image viewer which supports FITS files. You must check the DS9 first whether it is available with the SAMP.
    		--aladin : PyVI will use the Aladin as an image viewer which supports FITS and other file formats such as JPEG and PNG. You must check the Aladin first whether it is available with the SAMP. If both --ds9 and --aladin options are selected, both viewers are used in the PyVI.
    		--general : PyVI will use a general FITS image viewer including JSky, DS9, and Aladin. You must check the viewer first whether it is available with the SAMP. However, do not use --general with either --ds9 or --aladin together. This option is specially for FITS viwer showing only a single frame. Therefore, this option is appropriate for tools like JSky.
    		--spec : PyVI will use a spectrum viewer (e.g. SPLAT-VO, VOSpec). You must check the viewer first whether it is available with the SAMP.
    		--image_dir, --text_dir, --spec_dir : the directory names are optional
    		if the directory names are not given, the current working directory is a default directory.
    
  • If you do not need to use FITS files with versions 3.0 and 3.5, you do not need to install SAMPY. Yet, you still need the Python Image Library (PIL) and the TkInter library to view other image files supported by the PIL.
  • - Python code to produce multiple HTML files with objects list

    Check the code. You can use this simple script to produce multiple HTML files which show the list of objects with images and links to external web sites. The code also has simple functions to convert angles given in degree to sexagesimal formats. If you want to get only these functions, check coordinates_conversion.py below.

    - Python code to convert FITS to ordinary graphic files

    Please, check this link. If you need a simple way, please use fits2jpeg.sh given below.

    - Variability detection and period estimation

    Please, check this link.

    - Bash shell scripts to use IRAF for reducing dithered images (of NIR)

    You can find details about this package of shell scripts in this link.

    - Miscellaneous codes

  • coordinates_conversion.py : if you are looking for a short Python code to convert formats of RA/DEC coordinates, this code is exactly for that purpose.
  • plot_2d_max_L.py : it implements in Python the algorithm that finds the optimized histogram bins for 2D data. See arXiv:0807.4820. The code uses Numpy and Matplotlib for some parts. However, you can avoiid the required packages by modifying the code.
  • parser_of_YY.py : a simple Python code to separate each different age of the Yonsei-Yale isochrone into several files.
  • finding_chart.tar.gz : a collection of bash shell scripts to download DSS or SDSS images from SkyView and mark a target on jpeg images. This uses montage/convert of ImageMagick and DS9.
  • image_align.tar.gz : a collection of bash shell scripts that use IRAF and DS9 to align multiple FITS images to a given reference image.
  • fits2jpeg.sh : a simple bash shell script that uses DS9 to produce jpeg images for given FITS files. If you want to change a color map, you need to check command-line options of DS9.
  • Gal_cat_maker.py : a Python script to group target objects for an input catalog of observation fields.
  • Observation Help : a Python code that uses PyEphem. It calculates Az., Alt., Rise/Transit/Set time for given objects and date.
  • Curve of Growth : C code of calculating a curve of growth. (It is dependent on the GNU scientific library.)
  • Useful list or table

  • Matplotlib color maps
  • --------------------------------

    "Controlling complexity is the essence of software development." - Brian Kernigan

    --------------------------------