Fuse
Lightweight fuzzy-search, in JavaScript
Fuse.js | K. Risk - JavaScript Refined lightweight fuzzy-search, in javascript
I'm presently writing a filesystem. The statvfs (and even the statfs) structs contain a field specifying the maximum length of a name in that path. As PATH_MAX is defined in the pathconf manpage (getconf), this means it is defined on a per-directory basis (and thus, determined by the underlying filesystem). How does one specify this value?
Source: (StackOverflow)
FUSE is constantly(every 2 - 3 days) giving me this Transport endpoint is not connected error on my mount point and the only thing that seems to fix it is rebooting.
I currently have my mount points setup like this, I'm not sure what other details I should add here so let me know if I missed anything..
/dev/sdc1 /mnt/hdd2 ext4 defaults 0 0
/dev/sdb1 /mnt/hdd1 ext4 defaults 0 0
mhddfs#/mnt/hdd1,/mnt/hdd2 /data fuse defaults,allow_other 0 0
Source: (StackOverflow)
I'm implementing a FUSE filesystem intended to provide access via familiar POSIX calls to files which are actually stored behind a RESTful API. The filesystem caches files once they've been retrieved for the first time so that they are more readily available in subsequent accesses.
I'm running the filesystem in multi-threaded mode (which is FUSE's default) but finding that the getattr calls seem to be serialised, even though other calls can take place in parallel.
When opening a file FUSE always calls getattr first, and the client I'm supporting needs the file size returned by this initial call to be accurate (I don't have any control over this behaviour). This means that if I don't have the file cached I need to actually get the information via the RESTful API calls. Sometimes these calls happen over a high latency network, with a round trip time of approximately 600ms.
As a result of the apparent sequential nature of the getattr call, any access to a file which is not currently cached will cause the entire filesystem to block any new operations while this getattr is serviced.
I've come up with a number of ways to work round this, but all seem ugly or long-winded, really I just want the getattr calls to run in parallel like all the other calls seem to.
Looking at the source code I don't see why getattr should be behaving like this, FUSE does lock the tree_lock mutex, but only for read, and there are no writes happening at the same time.
For the sake of posting something simple in this question I've knocked up an incredibly basic implementation which just supports getattr and allows easy demonstration of the issue.
#ifndef FUSE_USE_VERSION
#define FUSE_USE_VERSION 22
#endif
#include <fuse.h>
#include <iostream>
static int GetAttr(const char *path, struct stat *stbuf)
{
std::cout << "Before: " << path << std::endl;
sleep(5);
std::cout << "After: " << path << std::endl;
return -1;
}
static struct fuse_operations ops;
int main(int argc, char *argv[])
{
ops.getattr = GetAttr;
return fuse_main(argc, argv, &ops);
}
Using a couple of terminals to call ls on a path at (roughly) the same time shows that the second getattr call only starts once the first has finished, this causes the second ls to take ~10 seconds instead of 5.
Terminal 1
$ date; sudo ls /mnt/cachefs/file1.ext; date
Tue Aug 27 16:56:34 BST 2013
ls: /mnt/cachefs/file1.ext: Operation not permitted
Tue Aug 27 16:56:39 BST 2013
Terminal 2
$ date; sudo ls /mnt/cachefs/file2.ext; date
Tue Aug 27 16:56:35 BST 2013
ls: /mnt/cachefs/file2.ext: Operation not permitted
Tue Aug 27 16:56:44 BST 2013
As you can see, the time difference from the two date outputs from before the ls differs only by one second, but the two from after the ls differs by 5 seconds, which corresponds to the delay in GetAttr function. This suggests that the second call is blocked somewhere deep in FUSE.
Output
$ sudo ./cachefs /mnt/cachefs -f -d
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.10
flags=0x0000000b
max_readahead=0x00020000
INIT: 7.8
flags=0x00000000
max_readahead=0x00020000
max_write=0x00020000
unique: 1, error: 0 (Success), outsize: 40
unique: 2, opcode: LOOKUP (1), nodeid: 1, insize: 50
LOOKUP /file1.ext
Before: /file1.ext
After: /file1.ext
unique: 2, error: -1 (Operation not permitted), outsize: 16
unique: 3, opcode: LOOKUP (1), nodeid: 1, insize: 50
LOOKUP /file2.ext
Before: /file2.ext
After: /file2.ext
unique: 3, error: -1 (Operation not permitted), outsize: 16
The above code and examples are nothing like the actual application or how the application is used, but demonstrates the same behaviour. I haven't shown this in the example above, but I've found that once the getattr call completes, subsequent open calls are able to run in parallel, as I would have expected.
I've scoured the docs to try and explain this behaviour and tried to find someone else reporting a similar experience but can't seem to find anything. Possibly because most implementations of getattr would be so quick you wouldn't notice or care if it was being serialised, or maybe because I'm doing something silly in the configuration. I'm using version 2.7.4 of FUSE, so it's possible this was an old bug that has since been fixed.
If anyone has any insight in to this it would be greatly appreciated!
Source: (StackOverflow)
I am trying to create bindings to the FUSE library using JNA, but I have hit a snag along the road. I have minimized the code as much as possible to make it digestible here.
The FUSE library comes with a few example filesystems written in C. The simplest of them is hello.c. The following is a minimized version of its code to simply a few prints in the filesystem functions:
hello.c:
/*
FUSE: Filesystem in Userspace
Copyright (C) 2001-2007 Miklos Szeredi <miklos@szeredi.hu>
This program can be distributed under the terms of the GNU GPL.
See the file COPYING.
gcc -Wall hello.c -o hello `pkg-config fuse --cflags --libs`
*/
#define FUSE_USE_VERSION 26
#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
static int hello_getattr(const char *path, struct stat *stbuf)
{
printf("getattr was called\n");
return 0;
}
static int hello_readdir(const char *path, void *buf, fuse_fill_dir_t filler, off_t offset, struct fuse_file_info *fi)
{
printf("readdir was called\n");
return 0;
}
static int hello_open(const char *path, struct fuse_file_info *fi)
{
printf("open was called\n");
return 0;
}
static int hello_read(const char *path, char *buf, size_t size, off_t offset, struct fuse_file_info *fi)
{
printf("read was called\n");
return 0;
}
static struct fuse_operations hello_oper = {
.getattr = hello_getattr,
.readdir = hello_readdir,
.open = hello_open,
.read = hello_read,
};
int main(int argc, char *argv[])
{
return fuse_main_real(argc, argv, &hello_oper, sizeof(hello_oper), NULL);
}
This can be compiled using gcc -Wall hello.c -o hello -D_FILE_OFFSET_BITS=64 -I/usr/include/fuse -pthread -lfuse -lrt -ldl
And invoked with ./hello.c -f /some/mount/point
The -f flag is to make it stay in the foreground so that you can see the printf()'s working.
All of this works well, you can see the printf()'s executing properly. I am trying to replicate the same thing in Java using JNA. Here is what I came up with:
FuseTemp.java:
import com.sun.jna.Callback;
import com.sun.jna.Library;
import com.sun.jna.Native;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;
public class FuseTemp
{
public static interface Fuse extends Library
{
int fuse_main_real(int argc, String[] argv, StructFuseOperations op, long size, Pointer user_data);
}
@SuppressWarnings("unused")
public static class StructFuseOperations extends Structure
{
public static class ByReference extends StructFuseOperations implements Structure.ByReference
{
}
public Callback getattr = new Callback()
{
public int callback(final String path, final Pointer stat)
{
System.out.println("getattr was called");
return 0;
}
};
public Callback readlink = null;
public Callback mknod = null;
public Callback mkdir = null;
public Callback unlink = null;
public Callback rmdir = null;
public Callback symlink = null;
public Callback rename = null;
public Callback link = null;
public Callback chmod = null;
public Callback chown = null;
public Callback truncate = null;
public Callback utime = null;
public Callback open = new Callback()
{
public int callback(final String path, final Pointer info)
{
System.out.println("open was called");
return 0;
}
};
public Callback read = new Callback()
{
public int callback(final String path, final Pointer buffer, final long size, final long offset, final Pointer fi)
{
System.out.println("read was called");
return 0;
}
};
public Callback write = null;
public Callback statfs = null;
public Callback flush = null;
public Callback release = null;
public Callback fsync = null;
public Callback setxattr = null;
public Callback getxattr = null;
public Callback listxattr = null;
public Callback removexattr = null;
public Callback opendir = null;
public Callback readdir = new Callback()
{
public int callback(final String path, final Pointer buffer, final Pointer filler, final long offset,
final Pointer fi)
{
System.out.println("readdir was called");
return 0;
}
};
public Callback releasedir = null;
public Callback fsyncdir = null;
public Callback init = null;
public Callback destroy = null;
public Callback access = null;
public Callback create = null;
public Callback ftruncate = null;
public Callback fgetattr = null;
public Callback lock = null;
public Callback utimens = null;
public Callback bmap = null;
public int flag_nullpath_ok;
public int flag_reserved;
public Callback ioctl = null;
public Callback poll = null;
}
public static void main(final String[] args)
{
final String[] actualArgs = { "-f", "/some/mount/point" };
final Fuse fuse = (Fuse) Native.loadLibrary("fuse", Fuse.class);
final StructFuseOperations.ByReference operations = new StructFuseOperations.ByReference();
System.out.println("Mounting");
final int result = fuse.fuse_main_real(actualArgs.length, actualArgs, operations, operations.size(), null);
System.out.println("Result: " + result);
System.out.println("Mounted");
}
}
The definition of the the fuse_operations struct can be found here.
This can be compiled using: javac -cp path/to/jna.jar FuseTemp.java
And invoked using java -cp path/to/jna.jar:. FuseTemp
jna.jar is available here.
The error that comes up is: fusermount: failed to access mountpoint /some/mount/point: Permission denied.
I am executing both programs as the same user with the same permissions on the same mountpoint folder, and I am in the fuse group. I am using:
- Linux kernel 3.0.0
- FUSE 2.8.4
- OpenJDK 1.6.0_23
- JNA 3.4.0
So my question is: What exactly is different between these two programs (hello.c and FuseTemp.java), and how to make them do the same thing?
Thanks in advance.
Edit: Here is some additional info.
Initial stat of the mountpoint:
File: `/some/mount/point'
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: 803h/2051d Inode: 540652 Links: 2
Access: (0777/drwxrwxrwx) Uid: ( 1000/ myusername) Gid: ( 1000/ myusername)
Output I get from running the Java program as regular user:
Mounting
fusermount: failed to access mountpoint /some/mount/point: Permission denied
Result: 1
Mounted
(program exits with return code 0)
After this, trying to execute stat gives the following error message:
stat: cannot stat/some/mount/point': Transport endpoint is not connected`
That is because the Java program isn't running anymore, so fuse cannot call its callbacks. To unmount, if I try fusermount -u /some/mount/point, I get:
fusermount: entry for /some/mountpoint not found in /etc/mtab
And if I try sudo fusermount -u /some/mount/point, the mountpoint is successfully unmounted and there is no output from fusermount. /etc/mtab is chmod'd 644 (-rw-r--r--) so my user can read it, but it doesn't contain /some/mount/point. After a successful unmount, the mountpoint is back to its old permissions (777 directory).
Now, running the java program as root:
Mounting
Result: 1
Mounted
(program exits with return code 0)
After that, stating /some/mount/point shows that is has not been modified, i.e. it is still a 777 directory.
I have also rewritten FuseTemp.java to include all Callbacks as Callbacks instead of Pointers. The behavior is the same, however.
I looked at fuse's source code and the error code 1 can be returned at multiple points throughout the execution. I will pinpoint where exactly is it failing on the fuse side and report back here.
Now for hello.c: running it as regular user, starting with the same permissions on /some/mount/point and passing it the arguments -f and /some/mount/point, the program doesn't print any output at first but keeps running. When running stat on the mountpoint, the program prints
getattr was called
like it should. stat returns an error, but that's simply because hello.c's getattr function doesn't give it any information, so no problems there. After executing fusermount -u /some/mount/point as regular user, the program exits with return code 0 and the unmount is successful.
Running it as root, starting with the same permissions on /some/mount/point and passing it the arguments -f and /some/mount/point, the program doesn't print any output at first but keeps running. When running stat on the mountpoint, I get a permission error because I am not root. When running stat on it as root, the program prints
getattr was called
like it should. Executing fusermount -u /some/mount/point as regular user yields
fusermount: entry for /some/mount/point not found in /etc/mtab
Executing fusermount as root, the program exits with return code 0 and the unmount is successful.
Source: (StackOverflow)
Currently I am developing an application using FUSE filesystem module in Linux (2.6 Kernel) in C language. Due to some programming error, the application crashes after mounting the filesystem. Since I am a novice developer in Linux/C environment. Could you please let me tell me possible options to debug such programs?
Source: (StackOverflow)
I'm working on a project that is being hosted on Amazon Web Services. The server setup consists of two EC2 instances, one Elastic Load Balancer and an extra Elastic Block Store on which the web application resides. The project is supposed to use S3 for storage of files that users upload. For the sake of this question, I'll call the S3 bucket static.example.com
I have tried using s3fs (https://code.google.com/p/s3fs/wiki/FuseOverAmazon), RioFS (https://github.com/skoobe/riofs) and s3ql (https://code.google.com/p/s3ql/). s3fs will mount the filesystem but won't let me write to the bucket (I asked this question on SO: How can I mount an S3 volume with proper permissions using FUSE). RioFS will mount the filesystem and will let me write to the bucket from the shell, but files that are saved using PHP don't appear in the bucket (I opened an issue with the project on GitHub). s3ql will mount the bucket, but none of the files that are already in the bucket appear in the filesystem.
These are the mount commands I used:
s3fs static.example.com -ouse_cache=/tmp,allow_other /mnt/static.example.com
riofs -o allow_other http://s3.amazonaws.com static.example.com /mnt/static.example.com
s3ql mount.s3ql s3://static.example.com /mnt/static.example.com
I've also tried using this S3 class: https://github.com/tpyo/amazon-s3-php-class/ and this FuelPHP specific S3 package: https://github.com/tomschlick/fuel-s3. I was able to get the FuelPHP package to list the available buckets and files, but saving files to the bucket failed (but did not error).
Have you ever mounted an S3 bucket on a local linux filesystem and used PHP to write a file to the bucket successfully? What tool(s) did you use? If you used one of the above mentioned tools, what version did you use?
EDIT
I have been informed that the issue I opened with RioFS on GitHub has been resolved. Although I decided to use the S3 REST API rather than attempting to mount a bucket as a volume, it seems that RioFS may be a viable option these days.
Source: (StackOverflow)
I am reading this sample code from FUSE:
http://fuse.sourceforge.net/helloworld.html
And I am having trouble understanding what the following snippet of code does:
static int hello_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
off_t offset, struct fuse_file_info *fi)
{
(void) offset;
(void) fi;
Specifically, the (void) "variable name" thing. I have never seen this kind of construct in a C program before, so I don't even know what to put into the Google search box. My current best guess is that it is some kind of specifier for unused function parameters? If anyone knows what this is and could help me out, that would be great. Thanks!
Source: (StackOverflow)
I'm trying to implement the FUSE filesystem. I am receiving this error:
cannot access MountDir: Transport endpoint is not connected
This the relevant parts of the program. There are two directories, MirrorDir and MountDir, that exist withing the same directory as all of the code. I am calling the program like this:
./myFS -o nonempty -o allow_other MirrorDir MountDir
Can anyone see what I am doing wrong?
static struct fuse_operations xmp_oper = {
.getattr = xmp_getattr,
.readdir = xmp_readdir,
.open = xmp_open,
.read = xmp_read,
};
int main(int argc, char *argv[]) {
int fuse_stat;
char* mirrorDir;
mirrorDir = malloc(sizeof(strlen(argv[argc-2]+1)));
if (mirrorDir == NULL) {
perror("main calloc");
abort();
}
// Pull the rootdir out of the argument list and save it in my internal data
mirrorDir = realpath(argv[argc-2], NULL);
argv[argc-2] = argv[argc-1];
argv[argc-1] = NULL;
argc--;
// turn over control to fuse
fprintf(stderr, "about to call fuse_main\n");
fuse_stat = fuse_main(argc, argv, &xmp_oper, mirrorDir);
fprintf(stderr, "fuse_main returned %d\n", fuse_stat);
return fuse_stat;
}
Source: (StackOverflow)
I just came across a VFS and a filesystem in userspace like FUSE.
Now, as far as I understand, it simulates a file system, so that an application can have a standard file system hierarchy. But I don't understand, why do we need a separate file system for that? Can't we just create a regular folder structure and place files which will be used by the app?
So, my questions are:
What is a VFS?
Can you give some real world examples, use cases where VFS is used.
What is the benefit of using a VFS?
Any Java based VFS?
Source: (StackOverflow)
I took fuse hello.c and modified the bottom to show what I am talking about. In my app I need to do things after my fuse FS is available. I also need another thread for IPC and keeping certain things up to date. Because fuse_main doesn't appear to return I threw it in its own thread.
When I comment out fuse_main the console shows A and B printed. However if I don't comment out fuse_main (which is in a different thread) only A is printed. How the heck is fuse stopping my main thread and how do I run code after FUSE does its thing?
#define FUSE_USE_VERSION 26
#include <fuse.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
#include <fcntl.h>
static const char *hello_str = "Hello World!\n";
static const char *hello_path = "/hello";
static int hello_getattr(const char *path, struct stat *stbuf)
{
int res = 0;
memset(stbuf, 0, sizeof(struct stat));
if (strcmp(path, "/") == 0) {
stbuf->st_mode = S_IFDIR | 0755;
stbuf->st_nlink = 2;
} else if (strcmp(path, hello_path) == 0) {
stbuf->st_mode = S_IFREG | 0444;
stbuf->st_nlink = 1;
stbuf->st_size = strlen(hello_str);
} else
res = -ENOENT;
return res;
}
static int hello_readdir(const char *path, void *buf, fuse_fill_dir_t filler,
off_t offset, struct fuse_file_info *fi)
{
(void) offset;
(void) fi;
if (strcmp(path, "/") != 0)
return -ENOENT;
filler(buf, ".", NULL, 0);
filler(buf, "..", NULL, 0);
filler(buf, hello_path + 1, NULL, 0);
return 0;
}
static int hello_open(const char *path, struct fuse_file_info *fi)
{
if (strcmp(path, hello_path) != 0)
return -ENOENT;
if ((fi->flags & 3) != O_RDONLY)
return -EACCES;
return 0;
}
static int hello_read(const char *path, char *buf, size_t size, off_t offset,
struct fuse_file_info *fi)
{
size_t len;
(void) fi;
if(strcmp(path, hello_path) != 0)
return -ENOENT;
len = strlen(hello_str);
if (offset < len) {
if (offset + size > len)
size = len - offset;
memcpy(buf, hello_str + offset, size);
} else
size = 0;
return size;
}
static struct fuse_operations hello_oper;
//modification starts below this line
#include<thread>
#include<unistd.h>
int main(int argc, char *argv[])
{
std::thread t([&]{
hello_oper.getattr = hello_getattr;
hello_oper.readdir = hello_readdir;
hello_oper.open = hello_open;
hello_oper.read = hello_read;
return fuse_main(argc, argv, &hello_oper, NULL);
});
printf("A\n");
sleep(5);
printf("B\n");
t.join();
}
Source: (StackOverflow)
I've written a small FUSE-based filesystem and now the only part's missing is that I want to register it with fstab(5) to auto-mount it on system startup and/or manually mount it with just mount /srv/virtual-db. How can I achieve this?
I know, I can just run /usr/bin/vdbfs.py /srv/virtual-db from some init script, but that's not exactly pretty.
I'm sorry because this may be not exactly a programming question, but it's highly related, as the packaging and deployment is still the programmer's job.
Source: (StackOverflow)
Can I write a custom userspace file system that can be run on non-rooted factory devices through the standard available utilities?
I am aware of the existence of fuse-android, however as far as I have understood, it requires a rooted device. If that is not the case, please do correct me.
The goal I am trying to achieve is create a 'fake' FS that actually is mounted to a file.
Source: (StackOverflow)
I'm using sshfs on macosx 10.7 in my every day job.
To install sshfs I have used macport: "port install fuse4x sshfs".
The todays error:
$ sshfs user@domain.com:/path myfolder
fuse4x client library version is incompatible with the kernel extension (kext='0.9.0', library='0.9.2').
Any tips welcomed. I don't understand, I have not updated the system.
Source: (StackOverflow)
I've found that the FUSE userspace library and kernel interface has been ported, since its inception on Linux, to many other systems, and presents a relatively stable API with a supposedly small surface area. If I wanted to author a filesystem in userspace, and I were not on Plan 9 or Hurd, I would think that FUSE is my best choice.
However, I am not going to use libfuse. This is partially because of pragmatism; using C is hard in my language of choice (Monte). It's also because I am totally uninterested in writing C support code, and libfuse's recommended usage is incompatible with Monte philosophy. This shouldn't be a problem, since C is not magical and /dev/fuse can be opened with standard system calls.
Going to look for documentation, however, I've found none. There is no documentation that I can find for the /dev/fuse ABI/API, and no stories of others taking this same non-C-bound route. Frustrating.
Does any kind of documentation exist on how to interact in a language-agnostic way with /dev/fuse and the FUSE subsystem of the kernel? If so, could you point me to it? Thanks!
Update: There exists go-fuse, which is in Go, a slightly more readable language than C. However, it does not contain any ABI/API documentation either.
Update: I notice that people have voted to close this. Don't worry, there is no need for that. I have satisfied myself that the documentation that I desire does not yet exist. I will write the documentation myself, publish it, and then link to it in an accepted answer. Hopefully the next person to search for this documentation will not be disappointed.
Source: (StackOverflow)