Category: Python

SING4DATASCIENCE/HSSingLog Process Summary

Overall Process

SING4DATASCIENCE/HSSingLog session contains number of processes.

  1. Generation of the playlist
  2. Sing session
  3. Result tabulation
  4. Metadata tabulation
  5. Result posting

This article covers processes and the tools used for the process.

Generation of the playlist

Playlist generation for the sing session is automated, and has following features.

  • Automatic lottery selection system
  • Fixed song system (currently applies to the 3 songs.)

Automatic lottery selection system uses the number of past entries and elapsed time since last session as the weighting element to simulate lottery for the songs to be included in a playlist. With this weighting, the system select songs with fewer number of sessions, and ones with longer time elapsed since last session.

This is implemented using PowerShell, with lottery logic as a binary, lottery process as a script. (Using PowerShell 7.)

Typical execution of the automatic lottery selection system

Sing session

After generation of the playlist is complete, the system generates singing directive sheet. This sheet contains order of songs to be sung, and also shows currently available metadata for these songs.

Singing directive sheet

Result tabulation

Once the singing session is complete, the results are tabulated. From the singing directive sheet, Title ID column is extracted, and is tabulated using tabulation sheet.

Tabulation sheet

Once this tabulation process is complete, using the Get-HSLCompiledFile cmdlet, verifications (to make sure scoring doesn’t have any irregularites) and combine the title information, which generates the results posted on GitHub.

Typical execution of the Get-HSLCompiledFile cmdlet

Metadata tabulation

When the song is missing metadata (either this is a new song, or ones that metadata is not yet filled in), these are entered into the singing directive sheet, and then the Update-HSLCsvData cmdlet is used to update. This cmdlet updates 3 metadata CSV files. (These 3 metadata files are authors, release date, and tempo.)

Typical execution of Update-HSLCsvData cmdlet

Result posting

For the posting data, a Python script is used. Chiefly, these scripts contains the following 2 functions. (These uses pandas, and Matplotlib.)

  • Generation of the text for the posts
  • Generation of the graph element for the posts
Typical execution of the result posting generation

Using Anaconda from PowerShell

After installing Anaconda, you can invoke conda init to initialize settings for PowerShell, including PowerShell Core. There is the problem of this approach as Anaconda is initialized every time PowerShell is launched. This is OK if you are using Python every time you launch PowerShell, however, with my case, it is often the case that I use PowerShell and no Python. This is inefficient.

I’ve decided to create a Cmdlet called Initialize-Conda. Here’s the content. You can replace C:/Path/to/Conda to your Anaconda location to adopt this to your use. (Available for Windows and Linux in Gist.)

function Initialize-Conda
{
	$CONDA_ROOT_DIR = "C:/Path/to/Conda" # Change this
        [System.Environment]::SetEnvironmentVariable("CONDA_EXE", "$CONDA_ROOT_DIR/Scripts/conda.exe", [System.EnvironmentVariableTarget]::Process)
	[System.Environment]::SetEnvironmentVariable("_CE_M", "", [System.EnvironmentVariableTarget]::Process)
	[System.Environment]::SetEnvironmentVariable("_CE_CONDA", "", [System.EnvironmentVariableTarget]::Process)
    [System.Environment]::SetEnvironmentVariable("_CONDA_ROOT", "$CONDA_ROOT_DIR", [System.EnvironmentVariableTarget]::Process)
        [System.Environment]::SetEnvironmentVariable("_CONDA_EXE", "$CONDA_ROOT_DIR/Scripts/conda.exe", [System.EnvironmentVariableTarget]::Process)

        Import-Module -Scope Global "$Env:_CONDA_ROOT/shell/condabin/Conda.psm1"
        conda activate base
}

Environment variable initialization appears bit long — this is because scope of the variable does not get passed outside of the script scope. Therefore, I had to use .NET’s SetEnvironmentVarible to initialize environmental variable for the scope of the process. (Under Linux, CONDA_EXE and _CONDA_EXE should be changed to $CONDA_ROOT_DIR/bin/conda)

It is also possible to use Add-CondaEnvironmentToPrompt to prefix the prompt with the current Conda environment. I’ve omitted this as I found this to be unreliable.

Here’s the corresponding PSD1 file. (Available from Gist.)

@{
    RootModule = 'Conda.psm1'
    ModuleVersion = '1.0.0.0'
    FunctionsToExport = @(
            'Initialize-Conda'
        )
    CmdletsToExport   = '*'
    VariablesToExport = '*'
    AliasesToExport   = '*'
    GUID = '23421dee-ca6f-4847-9c93-1268c629964a'
    Author = 'Hideki Saito'
    Description = 'Anaconda Initializer'
    PowerShellVersion = '6.0'
    CompatiblePSEditions = 'Core'
    Copyright = '(c) 2019 Hideki Saito. All rights reserved.'
    PrivateData = @{
        PSData = @{
            ProjectUri = ''
            ReleaseNotes = ''
        }
    }
}

Take these two files, Conda.psm1, and Conda.psd1, and place them under Documents\PowerShell\Modules\Conda (Under Linux, it’s ~/.local/share/powershell/Modules/Conda) and then you should be able to launch Initialize-Conda.