EzDevInfo.com

How to differ between external HDD and internal ones?

I want to know whether device is connected via USB (meaning it is a removable hard drive) or SATA (meaning it is an internal hard drive). That is how I get a list of devices

SP_DEVINFO_DATA volumeData;
volumeData.cbSize = sizeof (SP_DEVINFO_DATA);
SP_DEVICE_INTERFACE_DATA volumeInterfaceData;
volumeInterfaceData.cbSize = sizeof (SP_DEVICE_INTERFACE_DATA);
wchar_t buffer[1024];
PSP_DEVICE_INTERFACE_DETAIL_DATA volumeInterfaceDetail;
volumeInterfaceDetail = (PSP_DEVICE_INTERFACE_DETAIL_DATA)buffer;
volumeInterfaceDetail->cbSize = sizeof (SP_DEVICE_INTERFACE_DETAIL_DATA);
for (int j = 0; SetupDiEnumDeviceInterfaces (hVolumeInfo, 0, &GUID_DEVINTERFACE_VOLUME, j, &volumeInterfaceData); j++) {
    DWORD bufferPathSize = offsetof (SP_DEVICE_INTERFACE_DETAIL_DATA, DevicePath) + sizeof(TCHAR);
    SetupDiGetDeviceInterfaceDetail (
        hVolumeInfo,
        &volumeInterfaceData,
        volumeInterfaceDetail,
        bufferPathSize,
        &bufferPathSize,
        &volumeData
        ));
    <some actions here>
}

After such operations I get a kind of the following result for each connected volume:

volumeInterfaceDetail->DevicePath: "\\\\?\\storage#volume#{3ec3ba03-2789-11e4-8251-806e6f6e6963}#0000000015f00000#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}"

How can I detect an interface (USB, SATA) with which the considering device is connected? Or is there any other way to differ external and internal HDDs using WinAPI?

How to switch from PIO to DMA modes on a SATA Controller in code?

I am running windows 7. I need build an unattended/automated test to exercise some SATA drives. One of the test requirements is to exercise the drive with DMA in both states (enabled AND disabled).

To manually change this setting, you would:

   1) Open Device Manager.
   2) Expand "IDE ATA/ATAPI controllers".
   3) Pick one of the ATA Channels listed; right-click properties; Advanced tab.
   4) Toggle "Enable DMA".

As I said, I need to do this unattended. Any ideas on how it could be done? I have searched Windows Dev Center high and low, including the Device instance and setup stuff in the driver development kit. I can't find anything relevant there. I've also searched for registry keys that might do the job, but haven't found anything there either.

Writing a large file prevents large block DMA allocation

I'm working with a board with an ARM based processor running linux (3.0.35). Board has 1GB RAM and is connected to a fast SSD HD, and to a 5MP camera.

My goal is capturing high resolution images and write those directly to disk.

All goes well until I'm trying to save a very long video (over 1GB of data),

After saving a large file, it seems that I'm unable to reload the camera driver - it fails allocating a large enough DMA memory block for streaming (when calling dma_alloc_coherent()).

I narrowed it down to a scenario where Linux boots (when most of the memory is available), I then write random data into a large file (>1GB), and when I try to load the camera driver it fails.

To my question -

When I open a file for writing, write a large amount of data, and close the file, isn't the memory which was used for writing the data to HD supposed to be freed?

I can understand the why the memory becomes fragmented during the HD access, but when the transactions to the HD are completed - why is the memory still so fragmented that I cannot allocate 15MB of contiguous RAM?

Thanks

How to programmatically detect sata drive unplug in SuSE Linux?

Does anyone know of a method I can use to programmatically detect if a SATA hard drive has been unplugged? Our file system is mounted in READ-ONLY mode when we need to detect the removal of the drive. We noticed the other day that we were able to unplug a hard drive and everything continued to run without a hitch until the next time we attempted to read from a file on disk.

Direct control of ATA commands

I am working on hard drive analysis, and wanted to know if there is a way to directly control an ATA hard drive under windows. In short I want to do something like a packet sniffer, but for the ATA commands sent to/from the hard drive.

After that, I'd like to be able to write ATA commands directly the drive. If this is not possible under Windows then Linux is second choice.

Third choice is to make an FPGA that does all this under PC control.

Any ideas on if this is possible from standard APIs?

AHCI specification

I have a question regarding the AHCI spec:

Is the variable pDmaXferCnt in the port used when the transfer is a DMA write or read?

The description in the spec seems to indicate that it isn't, but the PRDs are used instead. But how does the HBA know how much data is to be sent or received to/from a SATA device? This information will be available in the sector count of a H2D FIS, but unless I have overlooked it there doesn't seem to be a register of variable that holds this value. The DX:transmit state also seems to indicate that pDmaXferCnt will have a set value, yet I can’t see where it would be set for a DMA read/write operation.

Thanks

When should WriteFile block when called on a HANDLE that was opened with FILE_FLAG_WRITE_THROUGH?

At work we run our unit tests regularly on several build machines with the help of Jenkins. One of the unit tests opens a file with FILE_FLAG_WRITE_THROUGH, and makes tens of thousands of WriteFile calls on this file. Why it does that is not relevant to this question. On just one of the build machines, this test timed out constantly. After some investigation, it turned out the root cause for this was that each call to WriteFile was blocking (the thread was switched out, and switched in later), making the test run substantially slower. Curiously enough, when the test was started manually on this machine, WriteFile did not block, and thus the test did not time out.

After hours of trial and error, I found a "workaround": on the affected machine if Jenkins was started with Task Scheduler, WriteFile blocked, if it was started manually or by placing a BAT file in the Startup folder, it did not. I thought I needed to investigate more, so I created a minimal repro, a small program that opens a file with FILE_FLAG_WRITE_THROUGH, and calls WriteFile on it with random data 10000 times. With this minimal repro, WriteFile blocks constantly, regardless of how I launch it. I'm really confused, so here I am asking for any explanation of this behavior. All the machines I performed tests on had SATA drives, with default settings (write caching and Windows write-cache flushing enabled).

First of all for me, it's not even 100% clear what FILE_FLAG_WRITE_THROUGH should do. Information on the internet states that a special flag is passed down to the device, asking it to flush its cache after the write. According to Raymond Chen (and a bunch of other people), this is mostly irrelevant nowadays, as the majority of SATA drives silently ignore this flag due to performance reasons. On the "should or should not block" side, this page clearly states that with FILE_FLAG_WRITE_THROUGH WriteFile should block:

The write call doesn't return until the data is written to the file.

Do you have any ideas why I'm seeing this strange behavior (inconsistently)? How can I investigate further?

How to detect when a new SATA drive is inserted using Java under Linux?

I am currently working on a project where I need to bulk import data into a Hadoop cluster. The data that needs to be imported into Hadoop resides on SATA disks. The data that needs to be imported are Encase forensics disks image (E01). I created an application that extract files from these images and import those files into Hadoop, this works fine but I need to start the process by hand.

I want to automate the process by hot-swapping the SATA drives and automatically start the extract process. I wrote my program in Java and it needs to be that way due to some external libraries I use to analyse the images. I searched the internet for a solution where Java is used to detect newly inserted drives but all I found where libraries that can detect usb drive or used udev rules to kick start the a process (I could use that but prefer a Java solution)

Does anyone know if something like what I have described exists? Or does anyone can point me into the right direction? It would be much appreciated!

Almost forgot...I use ubuntu 12.04 Server Edition as my operating system.

Does fsync(fd) work on a file created by external program?

I have a SATA hard disk with write cache disabled:

hdparm -W0 /dev/foo

I am operating on an ext4 partition with these mount options (amongst others):

data=ordered
auto_da_alloc

Linux kernel version is 2.6.32-5-686.

Now, I have an external program that I cannot modify, but that I know creates a file in the following way:

int fd = open(path);
write(fd, data, data_size);
close(fd);

I.e. it does not fsync before closing. So at this point, the data may probably be in RAM, somewhere in kernel/fs caches.

Note: the metadata is not a concern yet: the final metadata will be written and fsynced after I've made sure the data has reached the disk platters. The data itself is the concern.

So the question is, how can I help the data reach the actual disk platters?

I have thought of running this separate program afterwards:

int fd = open(path);
fsync(fd);
close(fd);

Will that help flush the data, or should I use a different approach?

Bypassing I/O scheduling and linux kernel page buffering

What I wanna trying to accomplish:

Developing an linux application in C language, that "exclusively" accesses a

PATA/SATA hard disk drive(HDD) to send ATA commands(in fact only those ATA commands that do not modify any byte on HDD accessed - eg. READ_SECTOR, IDENTIFY_DEVICE, SET_FEATURES, etc.).

By "exclusively", I mean that as soon as HDD is powered on(a custom hardware-which is a simple on-off switch, ensures that HDD is not powered on till the application is loaded and desires to do so), the first and only access to that HDD is only to my application. IOWs except my application, not even the linux kernel(including SCSI sub-system) nor any other application or process or human user will ever be able to access that HDD, unless when my application instructs/permits them to do so.

There is another requirement on my application: As the access to HDD is quite critical(in terms of control and not in terms of performance) in our app., so it is not desired that any I/O schedular is involved in the transactions done by application(performance on this HDD is not a constraint.). Also it is not desired that the data read from HDD is buffered by kernel buffer or page buffer. The application will read in block size of 512 bytes or a multiple of it only.

Now the problem that I'm facing is:

The SCSI sub-system resides below(and written to work with) I/O scheduler and kernel buffer or page buffer cache.

Although 'sg-driver' is provided by the SCSI subsystem to directly send commands(- Linux SCSI sub-system commands, not ATA or SCSI commands directly - which are then translated by libata to actual ATA commands. Am I right here?) to the HDD,but that is a I/O approach - you give i/p and get o/p, i.e. you have no control over the process of data transfer protocol (eg PIO, DMA and ATA status and Error registers, etc.) and device configuration(via Set Features ATA command.).

Also the error reporting mechanism must be sound and is specific to ATA protocol and not simply Linux SCSI subsystem error codes. IOWs my application need to have access to ATA error register and ATA status register on PATA/SATA HDDs.

What my application demands is exclusive control of HDD - eg. issuing a READ_SECTOR ATA cmd and then retrieving the data itself from HDD directly via reading i/o ports or via 'libata' with the above requirements must be satisfied.

What I can't do ?

I'm not going to write a PATA/SATA HBA device driver or every HBA available in market, as those are already included in kernel for libata.

What I learned till now ?

To accomplish the desired task, I may(or may not?) need to write a block device driver that interacts directly with VFS layer(or is there any way to bypass VFS even, so that my app. can directly communicate with this block driver) w/o involving/messing with kernel buffer or page buffer and I/O scheduler. This block driver will communicate directly with libata(bypassing SCSI subsystem upper layer ), which then communicates with PATA/SATA HBA driver.

Is it possible to write such a driver in a cpu architecture independent way?

Is it a feasible approach? If yes then would it affect I/O performance of other HDDs attached that are not accessed by my app. in this way. Do I need to write a system call over VFS(or bypassing it if possible)in this case for my application to communicate wih my block driver? Please enlighten me regarding this approach.

or is it possible for my block device driver to directly communicate with PATA/SATA HBA driver written for libata, But again would this approach affect I/O performance of other HDDs attached that are not accessed by my app. in this way. Also how would my application communicate with this block device driver?

Please enlighten me.

Also I wanna know about the same scenario for my application, with one difference - instead of PATA/SATA drives what if I have SCSI hard drives and its variants - specifically, SAS, Fibre channel and USB. And of course this time i'll not be using libata and ATA commands, but SCSI protocol commands.

Would u like to suggest a live cd distro as my application host, that contains PATA/SATA HBA libata drivers(- not for IDE sub-system as I will not be using it for it is depreciated now and hence might not be updated with regards for HBA drivers.) for most HBAs.

In short, what is the most direct way for an linux application to access a PATA/SATA or SCSI/SAS/Fibre Channel HDD.

I hope, I have provided sufficient info regarding my question, but if u wanna get more info or more clarification, please feel free to ask.

Update1 (on 27/6/2012):
With the useful discussion with Chris(see below) and my research, I reached the following conclusions:

1. that a readymade USB-to-PATA/SATA adapter will not solve my purpose, becoz it does not allow my app. or driver to change the data transfer mode(PIO vs DMA) on-the-fly, it does not allow my app. or driver to read ATA registers.

2. that a custom made USB-to-PATA/SATA adapter might help, but that will require either a embedded processor which need to implement ATA protocol, or an FPGA chip that implements the whole ATA protocol. But the embedded processor solution involves GPIO and is not good for SATA as it will require specialized transceivers, and the I/O performance will be an issue for both PATA and SATA- too slow for my application.

Such a adapter will talk to my linux-kernel driver (or via libusb) to my app. by a custom protocol that helps in communication b/w my app. and ATA protocol on embedded processor. In case of FPGA chip solution, I need to implement this protocol in FPGA itself alongwith the ATA protocol.

But at this point it is infeasible in terms of labour, time and money for me to implement FPGA solution and embedded processor solution. So I'm stuck with software only solution.

Finally, it seems that I'm probably going to have to duplicate and modify everything down to the hardware inteface layer to meet by requirements as said by Chris.

So, between VFS layer and HBA driver or libata layer, exactly how should I proceed. What things need to be implemented and what not?

Can someone throw light on this issue? Any ideas??

Update2 (on 1/7/2012):
I'm struggling with the issue. Is there someone on SO, who can enlighten me?

Configuring a 7-Series GTXE2 transceiver for Serial-ATA (Gen1/2/3)

Hello this will be an experts questions :) You should be familiar with the following topics

• Xilinx Multi-Gigabit-Transceivers (MGTs), especially the 7-Series GTX/GTH transceivers (GTXE2_CHANNEL)
• Serial-ATA Gen1, Gen2 and Gen3, especially Out-of-Band (OOB) communication

Question:

How should a GTXE2 be configured for Serial-ATA?

OOB signaling is not working neither RX_ElectricalIdle nor ComInit.

Introduction:

I implemented a SATA controller for my final bachelor project, which supports multiple vendor/device platforms (Xilinx Virtex-5, Altera Stratix II, Altera Stratix IV). Now it's time to port this controller to the next device family: Xilinx 7-Series devices, by name a Kintex-7 on a KC705 board.

The SATA controller has a additional abstraction layer in the physical layer, which is based on SAPIS and PIPE 3.0. So to port the SATA controller to a new device family, I have only to write a new transceiver wrapper for a GTXE2 MGT.

As of Xilinx's CoreGenerator doesn't support the SATA protocols in the CoreGen wizard, I started a transceiver project from scratch and applied all necessary settings as far as they are asked by the wizard. After that I copied the GTXE2_COMMON instantiation into my wrapper module, ordered the generics and ports into a meaning full schema.

As a third step I connected all unconnected ports (the wizards doesn't assign all values !!) to their default values (the default from UG476 or zero if not defined).

In step 4 I checked all generics and ports again against the UG476 if they are compatible to the SATA settings. After that I connected my wrapper ports to the MGT and inserted cross-clock modules if necessary.

As of the KC705 board has no 150 MHz reference clock, I program the Si570 to supply this clock as "ProgUser_Clock" after each board "bootup". The MGT is in powerdown mode (P2) while this reconfiguration. When the Si570 is stable, the MGT is powered up, the used Channel PLL (CPLL) locks after ca. 6180 clock cycles. This CPLL_Locked events releases the GTX_TX|RX_Reset wires, which cause a GTX_TX|RX_ResetDone event after additional 270|1760 cycles (all cycles @ 150 MHz -> 6,6 ns).

This behavior can be seen in chipscope, captured with a stable, uninterrupted auxiliary clock (200 MHz, slightly oversampled).

So the GXTE2 seams to be powered-up, operational and all clocks are stable.

GTXE2 ports to control the OOB signaling:

The MGT has several ports for OOB signaling. On TX these are:

• TX_ElectricalIdle - forces TX into electrical idle condition
• TX_ComInit - send a ComInit sequence
• TX_ComWake - send a ComWake sequence
• TX_ComFinish - sequence was send -> ready for next command

On RX:

• RX_ElectricalIdle - RX_n/TX_p are in electrical idle condition (low-level interface)
• RX_ComInit_Detected - a complete ComInit sequence was send
• RX_ComWake_Detected - a complete ComWake sequence was send

Detailed error desciption:

1. TX sends no OOB sequences if TX_ComInit is high for one cycle.
2. RX_ElectricalIdle is always high

Tests:

1. SATA loopback cable: cut a SATA cable and solder the apropriate wires ;) -- I'm using a special SFP to SATA adapter, which extends the KC705 with a SATA connector - http://shop.trioflex.ee/product.php?id_product=73
2. SMA loopback cables: I moved the MGT and connected the LVDS wires to the SMA jacks and installed 2 SMA cables as cross-over.
3. I programmed my old ML505 (Virtex-5) with onboard SATA connector to send ComInit sequences. The 2 boards are connected with a special SATA cross-over cable.
4. I connected a HDD with a partial stripped SATA cable to the KC705 (SFP2SATA adapter) and connected a 2.5 GSps scope (yes the signals are undersampled, but it's good to see bursts and idle periods...).

Experiences:

• Test 3 shows transmitted OOB sequences from Virtex-5 to Kintex-7 but the ChipScope trigger event does not occur - Rx_ElectricalIdle is still high.
• Test 4 shows no transmitted OOB sequences on the cable.

Should I post parts or the complete transceiver instanziation?

only the instance has ca. 650 lines :(

Please ask if you need more information, images, code, ... :)

Appendix:

Electrical idle means that the MGT drives both LVDS wires (TX_n/TX_p) with common mode voltage (V_cm) which is in range 0..2000 mV. If this condition is met, the common mode delta voltage is less than 100 mV, which is referred to as ElectricalIdle condition.

OOB-signaling means that the MGT transmits bursts of electrical idle and normal data symbols (D10.2 in 8b/10b notation) on the LVDS wires. SATA/SAS defines 3 OOB sequences call ComInit, ComWake, ComSAS which have different burst/idle durations. Host controllers and devices use these "Morse signals" to establish a link.

ATA PASS THROUGH DIRECT on SAS BUS in Windows XP

I want to send a "ATA Pass through Direct" command to a drive that is on the SAS(SATA attached SCSI) bus. The drive is SATA drive, in Windows XP(x86) envirnoment. I hoped there was a STP(SATA Tunneled Protocol) layer that would translate the ATA command to STP to send through SCSI bus to my SATA drive. However, when sending "ATA Pass through direct" command the Win32 API DeviceIOControl(), status returns an error "wrong functions". Does STP layer exists in Windows XP, it must, Win32 API Readfile()/Writefile() work. But how do I get "ATA Pass through direct" command to work with the Win32 API on SAS bus?

LBA level disk access

What's the best way to issue disk commands (read k blocks from lba n etc.) to a SATA disk? The choice of OS does not matter. I also need to be able to issue ATA commands. I would rather do this through a tool/application than use ioctls programmatically.

CRC calculation for Serial ATA (SATA)

After reading a lot I came to know that there is no SINGLE method of calculating CRC. I need method/algorithm/VHDL code for calculating CRC specifically for Serial ATA (SATA)

write on track / head / cylinder on a magentic disk

I am developing a high performance database and I want to write directly to a specific track/cylinder with multiple heads at the same time. How is this done in Linux with SATA disks through the SATA bus? Any docs, examples you could recommend?