Simple Copy Files & Folders Script in Python

As I’ve mentioned before, I use Microsoft’s Windows Home Server as my home’s central data store. Recently, I upgraded from the trial to the full version. Since I was going to upgrade the installation, I figured that I might as well (slightly) upgrade the server itself. As such, I wanted to copy off the data from the hard drives before doing a format & reinstall of WHS. To do so, I accessed the “hidden” DE folder located on each of my WHS hard drives and copied the data to my local machine. When the data was transferred, I formatted the drive and re-added it to my new WHS install. However, when I started to transfer the data back, I noticed that many folders were scattered across multiple hard drives. Rather than manually merging the folders back together, I decided to write this simple Python script.

[2012-08-14 update: I’ve added a Python 3 compatible script (minor changes needed)]

Python 2

import os
import shutil
sourcePath = r'E:\some\path'
destPath = r'E:\some\other\path'
for root, dirs, files in os.walk(sourcePath):

    #figure out where we're going
    dest = destPath + root.replace(sourcePath, '')

    #if we're in a directory that doesn't exist in the destination folder
    #then create a new folder
    if not os.path.isdir(dest):
        os.mkdir(dest)
        print 'Directory created at: ' + dest

    #loop through all files in the directory
    for f in files:

        #compute current (old) & new file locations
        oldLoc = root + '\\' + f
        newLoc = dest + '\\' + f

        if not os.path.isfile(newLoc):
            try:
                shutil.copy2(oldLoc, newLoc)
                print 'File ' + f + ' copied.'
            except IOError:
                print 'file "' + f + '" already exists'

Python 3

import os
import shutil
sourcePath = r'D:\some\path'
destPath = r'D:\some\other\path'
for root, dirs, files in os.walk(sourcePath):

    #figure out where we're going
    dest = destPath + root.replace(sourcePath, '')
    
    #if we're in a directory that doesn't exist in the destination folder
    #then create a new folder
    if not os.path.isdir(dest):
        os.mkdir(dest)
        print('Directory created at: ' + dest)

    #loop through all files in the directory
    for f in files:

        #compute current (old) & new file locations
        oldLoc = root + '\\' + f
        newLoc = dest + '\\' + f

        if not os.path.isfile(newLoc):
            try:
                shutil.copy2(oldLoc, newLoc)
                print('File ' + f + ' copied.')
            except IOError:
                print('file "' + f + '" already exists')

–Adam

27 thoughts on “Simple Copy Files & Folders Script in Python

  1. acarter Post author

    Apparently, newer versions of Windows handles the merging of files and folders automatically, so this script is completely unnecessary! Oh well…

  2. luthar

    Hi .. I’m using your code to copy files from another PC on intranet to 1 PC. It looks pretty useful for me (thks for sharing your code) but except that it creates a file “khq.file” that doesn’t exist in my source file. when i want to recopy again it prompts the error. (I intend to backup the data from another PC). Is there a way to avoid for this?

    thks a lot for ya program

  3. acarter Post author

    Luthar,

    The script shouldn’t create any additional files. Perhaps “khq.file” exists in your source directory but is hidden in Windows Explorer? What is the exact error that you’re getting?

    –Adam

  4. Chris

    If your using python 2.6 or higher you could use a built it method to copy over an entire directory tree:

    shutil.copytree(SOURCE, DESTINATION)

  5. Farrah Dorsey

    This is some great code. Questin is can I use this code to read mutiple images in a dir and put them into a doc archive Postgresql tables? I have an idea using your code.

  6. acarter Post author

    Farrah, I don’t see why not. On your tree walk, just limit the files you search for based on extension (e.g. “png”) and then write an import query into your database.

  7. sbrinson

    Thanks for posting your code, it does exactly what I was searching for. I tried to enhance it by allowing the Source and Destination paths to be determined once run, but apparently I’m not versed enough in python to get it working. Any suggestions? When I define the paths statically it works great, but once I switch them to raw_input the script runs but transfers nothing.

  8. acarter Post author

    When entering the path using raw_input(), are you by chance using the ‘\’ character? If so, you’re probably creating illegal escape sequences. Instead, try using ‘/’.

    Example:
    (bad) “d:\temp”
    (good) “d:/temp”

  9. Renu

    Hai,

    I am very new to programming specially python.I have been given a task.i have to define a function which looks like:

    my_func(input_db, output_directory).

    it has to be robust and:

    Check if there are any matching NS**** or T**** files existing.

    Only for those files the function has to be executed!

    All results should be stored inside a user-definable directory. The function should print out how many data sets
    were found and how many data sets were processed.
    can any one help please? :)its urgent

  10. Robert

    I’d like to use this as a basis for esentially automatic filing. Daily, moving 100+ number specific files into number specific coresponding folders.

    Do you think I could adapt this code for that? If so, what should I change?

  11. acarter Post author

    Robert, you should be able to use this script as-is assuming you change the source and destination directories. Also, sorry for taking so long to respond.

  12. Pingback: Simple Copy Files & Folders Script in Python - Illisus.com

  13. Alex Knoll

    Hello Adam,
    my name is Alex and i’m from germany, so sorry first for my bad english…
    I’m verry thank you to post this script! On my lokal pc it works verry well, but if i try to change the surce path to my fileserver, it runns in a endless loop…
    Can you explain me what i have to change to get it working, please?

    THX

    regards
    Alex

  14. acarter Post author

    Alex,

    Are you trying to use a network path (e.g. “\\fileserver\music”)? If so, have you tried to map the network path to a local drive letter (e.g. “Z:\”)? Also, what operating system are you running this on?

    Thanks,
    –Adam

  15. Alex Knoll

    Hai,
    wow that was fast…^^
    Ok i try to explain it…
    Yes the server path is \\blabla.net/foldername/….
    No i’ve not tried to map the netpath…sounds like i should do this…ok…todo list…
    My fileserver uses WinXp pro 32bit,my lokal pc Win7x64Ultimate….

    Alex

  16. Ismail Rizvi

    Well this is just the code I wanted I work in android qtp and use python for some reasons well there was a testing scenario where every time I extract new libraries needs to be copied into my winscript folder it does the same i don’t have to worry about copying again and again..

    Thanks again.

  17. snigdha

    Hi Alex,

    I am new to Python and I have been trying to copy files from my network folder to my personal folder so I can analyze the data . But I just need the .txt files and not all the files , can you please suggest how I would be able to do that.

    Thanks,
    Snigdha

  18. acarter Post author

    Before doing the copy, you could so something like:


    name, extension = os.path.splitext(f)
    if extension == '.txt':
    #make the copy

  19. ans

    Hi, I have try your code. When I copy from one directory to another directory, it only copy the content of the folder (ex: test.txt) is in the folder. It will not copy the folder as well. How can I copy the file and the folder as well? thanks.

  20. Adam Mann

    Hi,
    I think this is exactly what I am looking for.

    I want to use a Raspberry Pi to auto copy files from my CF/SD card reader onto a USB HDD when out in the field (as a backup) and then when back in the office transfer them from the RPi to a NAS (or something like this) ready to start the edit.

    these cards can be up to 32gb so Im guessing this will just keep going till it is completed ? and also is there any verification or check-sum to make sure it is an exact copy ?

    Adam

  21. acarter Post author

    @ans, it should be making the folder with the lines:
    if not os.path.isdir(dest):
    os.mkdir(dest)

    @Adam, I’ve used this script to copy 100+ GB between hard drives, so you should be good there. Regarding checkusms, I’m not performing any, but it wouldn’t be too hard to add a check after each copy (see http://stackoverflow.com/questions/3431825/generating-a-md5-checksum-of-a-file). However, I wouldn’t be surprised if performing checksums on 32GB worth of files will significantly increase copy time.

  22. Adam Mann

    Hi Adam

    With the checksums it was more of a thought but I can try it out and see how it goes, I was wondering if you ever thought about building this into a APP that had a simple GUI, as I looked into this and well a little to complex for me TBH, I have something in mind but dont even know where to start building a GUI as this will eventually be running on a RPi with a 2.8″ touch screen (hopefully)

  23. acarter Post author

    Is the GUI touch? Does it provide an on-screen keyboard? Adding a GUI for a small screen would be an interesting design challenge. Unfortunately, it’s probably one that I don’t have much time for :(

  24. Adam Mann

    Hi Adam

    yes the GUI would be touch, No keyboard as it will just do a direct copy of whats on the ROOT of the card, yea I was thinking of having like 3/4 buttons

    https://dl.dropboxusercontent.com/u/41964/UI-Dev.docx

    button 1: CARD > USB HDD
    button 2: USB HDD > NAS
    button 3: WIPE HDD
    button 4: SHUTDOWN RPi – possibly

    it is a pity you don’t have the time as I don’t have the skills :( I might try to build something but I haven’t got a clue where to start.

    Adam

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>