Anyone using Python?

  • xsevensinzx

    One Orange Chip

    Points: 25531

    Man, I always come by here and never see anything. I'm always tooting how great Python is in these forums so I figured I would make the first post here.

    Question

    Anyone actually using Python for machine learning or well, anything relating to analytics, BI and statistics?

    I have for some time, but only for data ingestion (e.g.: API). I have been trying to sharpen up my scripting skills to include more machine learning techniques with Python. I have a growing library of Python scripts that have been collected from a number of data science & machine learning courses with a focus on Python over R.

    Lately, I've been doing hacker rank problems to also sharpen my skills. I ran into this problem where I think Python solves the challenge, but not fast enough to meet the requirements on larger data sets. If anyone has any insights, please feel free to jump in below.

    Problem I've Been Working On

    Given two numbers NN and MM. NN indicates the number of elements in the array and MM indicates number of queries. You need to perform two types of queries on the array. You are given MM queries. Queries can be of two types, type 1 and type 2.

      Type 1 queries are represented as 1 i j : Modify the given array by removing elements from ii to jj and adding them to the front.

      Type 2 queries are represented as 2 i j : Modify the given array by removing elements from ii to jj and adding them to the back.

    Your task is to simply print |A[1]−A[N]||A[1]−A[N]| of the resulting array after the execution of MM queries followed by the resulting array. Note While adding at back or front the order of elements is preserved.

    Input Format

    First line consists of two space-separated integers, NN and MM. Second line contains NN integers, which represent the elements of the array. MM queries follow. Each line contains a query of either type 1 or type 2 in the form type i j

    Sample Input

    8 4

    1 2 3 4 5 6 7 8

    1 2 4

    2 3 5

    1 4 7

    2 1 4

    Explanation

    Given array is {1,2,3,4,5,6,7,8}

    After execution of query 1 2 4, the array becomes {2,3,4,1,5,6,7,8}

    After execution of query 2 3 5, the array becomes {2,3,6,7,8,4,1,5}

    After execution of query 1 4 7, the array becomes {7,8,4,1,2,3,6,5}

    After execution of query 2 1 4, the array becomes {2,3,6,5,7,8,4,1}

    Now |A[1]−A[N]||A[1]−A[N]| is |(2−1)||(2−1)| i.e. 11 and the array is 23657841

    Data

    input_data.txt attached.

    My Code Submission

    This is too slow.. 🙁

    def array_query(s, t, i, j):

    """ Type 1 Query that modifies the given array by removing elements from i to j and adding them to the front.

    :param i: N starting number for range

    :param j: N ending number for range

    :return : New array with the chopped number ranges specified from i and j.

    """

    if t == 1:

    s[:0] = s[i-1:j]

    del(s[i+(j-i):j+(j-i+1)])

    elif t == 2:

    s += s[i-1:j]

    del(s[i-1:j])

    return s

    if __name__ == "__main__":

    input_size = map(int, raw_input().split()) # Array size, Total Queries

    source_array = map(int, raw_input().split()) # Input array

    for case in range(0, input_size[1]):

    test_case = map(int, raw_input().split())

    array_query(source_array, # a[ ] = Input Array

    test_case[0], # q[0] = Query type

    test_case[1], # q[1] = i range (Min)

    test_case[2]) # q[2] = j range (Max)

    print abs(source_array[0] - source_array[input_size[0] - 1])

    print ' '.join(map(str, source_array))

  • Steve Jones - SSC Editor

    SSC Guru

    Points: 714623

    Interesting. I'll have to play with this. No time now, but I've done some Python, and have been working with the SDTIG on their Python series. I've gotten behind, but they had an interesting machine learning segment, with the last meeting this Wed.

    http://www.meetup.com/San-Diego-Technology-Immersion-Group-SDTIG/

    I think I'd do something similar to your approach. String slicing makes sense, though probably not super fast.

    Do you mean slow in terms of lots of input rows, wide strings, or both?

  • xsevensinzx

    One Orange Chip

    Points: 25531

    Steve Jones - SSC Editor (4/24/2016)


    Interesting. I'll have to play with this. No time now, but I've done some Python, and have been working with the SDTIG on their Python series. I've gotten behind, but they had an interesting machine learning segment, with the last meeting this Wed.

    http://www.meetup.com/San-Diego-Technology-Immersion-Group-SDTIG/

    I think I'd do something similar to your approach. String slicing makes sense, though probably not super fast.

    Do you mean slow in terms of lots of input rows, wide strings, or both?

    This is slow because the bigger the phone book, the longer it takes so to speak. I need a better approach that scales to meet the 10 second minimum requirement. The attached data actually completes within 12 seconds on my machine. It's not that slow, but not fast enough. The below recommendation is one way to tackle the problem.

    The recommended approach is by doing a Treap. This is similar to a binary search tree, but a self-balancing binary search tree. I've seen examples in C++, but not too much in Python without modules. Trying to do this without a module. Guess I can manually do it.

    Treap

  • Bosko Vukov

    Old Hand

    Points: 339

    Hi, i think you can speed up your list "slice and dice" if you convert list to (byte) string using struct module and pack/unpack.

    from struct import *

    def string_query(s, t, i, j):

    """ Type 1 Query that modifies the given array by removing elements from i to j and adding them to the front.

    :param i: N starting number for range

    :param j: N ending number for range

    :return : New array with the chopped number ranges specified from i and j.

    """

    if t == 1:

    return s[ 4*(i-1): 4*j ] + s[ : 4*(i-1) ] + s[ 4*j : ]

    else:

    return s[ : 4*(i-1) ] + s[ 4*j : ] + s[ 4*(i-1): 4*j ]

    if __name__ == "__main__":

    ifl = open('input_data.txt','r')

    isz = map(int, ifl.readline().split()) # Array size, Total Queries

    sa = map(int, ifl.readline().split()) # Input array

    lsa = len(sa)

    sa = ''.join( map( lambda x: pack('i', x ), sa ))

    for case in range(0, isz[1]):

    tc = map(int, ifl.readline().split())

    sa = string_query( sa, # a[ ] = Input Array

    tc[0], # q[0] = Query type

    tc[1], # q[1] = i range (Min)

    tc[2]) # q[2] = j range (Max)

    sa = unpack( 'i' * lsa, sa )

    print abs(sa[0] - sa[isz[0] - 1])

    print ' '.join(map(str, sa))

  • xsevensinzx

    One Orange Chip

    Points: 25531

    They are strict on module usage. Don't think I can use it. Let's see.

  • xsevensinzx

    One Orange Chip

    Points: 25531

    Well then, I guess struct was good to go. Amazing how something as simply taking an approach to pack and unpack the data allowed me to reduce the runtime below 10 seconds on all 27 test cases.

    To compare the results. I went from 7.31 seconds on the last good test case with the previous code to over 10 seconds to fail the rest of the test cases.

    Using your suggestion, I was able to solve all the test cases under 10 seconds, with the longest running time 8.39 seconds.

    😀

  • Bosko Vukov

    Old Hand

    Points: 339

    🙂 https://www.hackerrank.com/challenges/array-and-simple-queries ...

    I hope you get that job.

  • xsevensinzx

    One Orange Chip

    Points: 25531

    Bosko Vukov (4/27/2016)


    🙂 https://www.hackerrank.com/challenges/array-and-simple-queries ...

    I hope you get that job.

    Pssh, no job attached to this. I do these for fun and stream them on TwitchTv to sharpen my skills! 😉

  • Steve Jones - SSC Editor

    SSC Guru

    Points: 714623

    Simpler, but I've done some of these: http://exercism.io/

    Python is built to use modules. Trying to not use them, in some sense, defeats the purpose. I guess it gains algorithmic practice, but perhaps leads you to spend time on items that could better be done in other ways.

    I guess it's a balance of some sort.

  • xsevensinzx

    One Orange Chip

    Points: 25531

    Steve Jones - SSC Editor (4/27/2016)


    Simpler, but I've done some of these: http://exercism.io/

    Python is built to use modules. Trying to not use them, in some sense, defeats the purpose. I guess it gains algorithmic practice, but perhaps leads you to spend time on items that could better be done in other ways.

    I guess it's a balance of some sort.

    It's likely because a lot of modules do all the work for you. They are trying to restrict that and see what you can do with the bare essentials for learning purposes. Having to always lean on a module will surely hinder you in solving these complex problems.

  • Steve Jones - SSC Editor

    SSC Guru

    Points: 714623

  • primitivefuture2006

    Old Hand

    Points: 358

    Hi Steve,

    I notice you link a lot of your "Python Question of the Day" gets linked to Real Python's website as a reference. Have your studied their tutorials in the past, and have they been helpful? I am considering getting their book. Thanks!

  • Steve Jones - SSC Editor

    SSC Guru

    Points: 714623

    I don't link to any particular one. I've used a few different ones, but probably learned the most from DiveIntoPython.com and that book

Viewing 13 posts - 1 through 13 (of 13 total)

You must be logged in to reply to this topic. Login to reply