Skip to content

Commit a5ef33a

Browse files
jsignjsign
authored
Stage folders support (textileio#559)
* pow add folder support Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * fixes Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * update docs Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * update readme Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * add authentication, lints Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * retrieve folders and update docs Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * fix flag name Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * better msg errors Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * move logic to ffs client Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com> * remove extra stop Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
1 parent 79533a0 commit a5ef33a

10 files changed

Lines changed: 247 additions & 50 deletions

File tree

‎README.md‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -75,6 +75,7 @@ The binary will be placed automatically in `$GOPATH/bin` which in general is in
7575
You can read the [generated CLI docs](https://github.com/textileio/powergate/blob/master/cli-docs/pow/pow.md) in this repo, or run `pow` with the `--help` flag to see the available commands:
7676

7777
```
78+
7879
$ pow --help
7980
A client for storage and retreival of powergate data
8081
@@ -83,22 +84,18 @@ Usage:
8384
8485
Available Commands:
8586
asks Provides commands to view asks data
86-
deal Interactive storage deal
87-
deals Provides commands to manage storage deals
87+
faults Provides commands to view faults data
8888
ffs Provides commands to manage ffs
8989
health Display the node health status
9090
help Help about any command
91-
init Initializes a config file with the provided values or defaults
9291
miners Provides commands to view miners data
9392
net Provides commands related to peers and network
9493
reputation Provides commands to view miner reputation data
95-
faults Provides commands to view faults data
9694
wallet Provides commands about filecoin wallets
9795
9896
Flags:
99-
--config string config file (default is $HOME/.powergate.yaml)
10097
-h, --help help for pow
101-
--serverAddress string address of the powergate service api (default "/ip4/127.0.0.1/tcp/5002")
98+
--serverAddress string address of the powergate service api (default "127.0.0.1:5002")
10299
103100
Use "pow [command] --help" for more information about a command.
104101
```
@@ -140,7 +137,7 @@ make build-powd
140137
You can run the `-h` flag to see the configurable flags:
141138
```bash
142139
$ powd -h
143-
Usage of ./powd:
140+
Usage of powd:
144141
--autocreatemasteraddr Automatically creates & funds a master address if none is provided
145142
--debug Enable debug log level in all loggers.
146143
--devnet Indicate that will be running on an ephemeral devnet. --repopath will be autocleaned on exit.
@@ -149,14 +146,14 @@ Usage of ./powd:
149146
--grpchostaddr string gRPC host listening address. (default "/ip4/0.0.0.0/tcp/5002")
150147
--grpcwebproxyaddr string gRPC webproxy listening address. (default "0.0.0.0:6002")
151148
--ipfsapiaddr string IPFS API endpoint multiaddress. (Optional, only needed if FFS is used) (default "/ip4/127.0.0.1/tcp/5001")
149+
--ipfsrevproxyaddr string IPFS reverse proxy listen address (default "127.0.0.1:6003")
152150
--lotushost string Lotus client API endpoint multiaddress. (default "/ip4/127.0.0.1/tcp/1234")
153151
--lotusmasteraddr string Existing wallet address in Lotus to be used as source of funding for new FFS instances. (Optional)
154152
--lotustoken string Lotus API authorization token. This flag or --lotustoken file are mandatory.
155153
--lotustokenfile string Path of a file that contains the Lotus API authorization token.
156154
--maxminddbfolder string Path of the folder containing GeoLite2-City.mmdb (default ".")
157155
--repopath string Path of the repository where Powergate state will be saved. (default "~/.powergate")
158156
--walletinitialfund int FFS initial funding transaction amount in attoFIL received by --lotusmasteraddr. (if set) (default 4000000000000000)
159-
pflag: help requested
160157
```
161158
162159
We'll soon provide better information about Powergate configurations, stay tuned! 📻

‎api/client/ffs.go‎

Lines changed: 96 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,17 @@ import (
44
"context"
55
"fmt"
66
"io"
7+
"net/http"
8+
"os"
9+
"strings"
710
"time"
811

912
cid "github.com/ipfs/go-cid"
13+
files "github.com/ipfs/go-ipfs-files"
14+
httpapi "github.com/ipfs/go-ipfs-http-client"
15+
"github.com/ipfs/interface-go-ipfs-core/options"
16+
ipfspath "github.com/ipfs/interface-go-ipfs-core/path"
17+
"github.com/multiformats/go-multiaddr"
1018
"github.com/textileio/powergate/deals"
1119
"github.com/textileio/powergate/ffs"
1220
"github.com/textileio/powergate/ffs/api"
@@ -371,6 +379,24 @@ func (f *FFS) Remove(ctx context.Context, c cid.Cid) error {
371379
return err
372380
}
373381

382+
// GetFolder retrieves to outputDir a Cid which corresponds to a folder.
383+
func (f *FFS) GetFolder(ctx context.Context, ipfsRevProxyAddr string, c cid.Cid, outputDir string) error {
384+
token := ctx.Value(AuthKey).(string)
385+
ipfs, err := newDecoratedIPFSAPI(ipfsRevProxyAddr, token)
386+
if err != nil {
387+
return fmt.Errorf("creating decorated IPFS client: %s", err)
388+
}
389+
n, err := ipfs.Unixfs().Get(ctx, ipfspath.IpfsPath(c))
390+
if err != nil {
391+
return fmt.Errorf("getting folder DAG from IPFS: %s", err)
392+
}
393+
err = files.WriteTo(n, outputDir)
394+
if err != nil {
395+
return fmt.Errorf("saving folder DAG to output folder: %s", err)
396+
}
397+
return nil
398+
}
399+
374400
// Get returns an io.Reader for reading a stored Cid from the Hot Storage.
375401
func (f *FFS) Get(ctx context.Context, c cid.Cid) (io.Reader, error) {
376402
stream, err := f.client.Get(ctx, &rpc.GetRequest{
@@ -463,7 +489,7 @@ func (f *FFS) Close(ctx context.Context) error {
463489
return err
464490
}
465491

466-
// Stage allows you to temporarily cache data in the Hot layer in preparation for pushing a cid storage config.
492+
// Stage allows to temporarily stage data in the Hot Storage in preparation for pushing a cid storage config.
467493
func (f *FFS) Stage(ctx context.Context, data io.Reader) (*cid.Cid, error) {
468494
stream, err := f.client.Stage(ctx)
469495
if err != nil {
@@ -500,6 +526,36 @@ func (f *FFS) Stage(ctx context.Context, data io.Reader) (*cid.Cid, error) {
500526
return &cid, nil
501527
}
502528

529+
// StageFolder allows to temporarily stage a folder in the Hot Storage in preparation for pushing a cid storage config.
530+
func (f *FFS) StageFolder(ctx context.Context, ipfsRevProxyAddr string, folderPath string) (cid.Cid, error) {
531+
ffsToken := ctx.Value(AuthKey).(string)
532+
533+
ipfs, err := newDecoratedIPFSAPI(ipfsRevProxyAddr, ffsToken)
534+
if err != nil {
535+
return cid.Undef, fmt.Errorf("creating IPFS HTTP client: %s", err)
536+
}
537+
538+
stat, err := os.Lstat(folderPath)
539+
if err != nil {
540+
return cid.Undef, err
541+
}
542+
ff, err := files.NewSerialFile(folderPath, false, stat)
543+
if err != nil {
544+
return cid.Undef, err
545+
}
546+
defer func() { _ = ff.Close() }()
547+
opts := []options.UnixfsAddOption{
548+
options.Unixfs.CidVersion(1),
549+
options.Unixfs.Pin(false),
550+
}
551+
pth, err := ipfs.Unixfs().Add(context.Background(), files.ToDir(ff), opts...)
552+
if err != nil {
553+
return cid.Undef, err
554+
}
555+
556+
return pth.Cid(), nil
557+
}
558+
503559
// ListPayChannels returns a list of payment channels.
504560
func (f *FFS) ListPayChannels(ctx context.Context) ([]ffs.PaychInfo, error) {
505561
resp, err := f.client.ListPayChannels(ctx, &rpc.ListPayChannelsRequest{})
@@ -748,3 +804,42 @@ func fromRPCRetrievalDealRecords(records []*rpc.RetrievalDealRecord) ([]deals.Re
748804
}
749805
return ret, nil
750806
}
807+
808+
func newDecoratedIPFSAPI(ipfsRevProxyAddr, ffsToken string) (*httpapi.HttpApi, error) {
809+
ipport := strings.Split(ipfsRevProxyAddr, ":")
810+
if len(ipport) != 2 {
811+
return nil, fmt.Errorf("ipfs addr is invalid")
812+
}
813+
cm, err := multiaddr.NewComponent("ip4", ipport[0])
814+
if err != nil {
815+
return nil, err
816+
}
817+
cp, err := multiaddr.NewComponent("tcp", ipport[1])
818+
if err != nil {
819+
return nil, err
820+
}
821+
ipfsMaddr := cm.Encapsulate(cp)
822+
customClient := http.DefaultClient
823+
customClient.Transport = newFFSHeaderDecorator(ffsToken)
824+
ipfs, err := httpapi.NewApiWithClient(ipfsMaddr, customClient)
825+
if err != nil {
826+
return nil, err
827+
}
828+
return ipfs, nil
829+
}
830+
831+
type ffsHeaderDecorator struct {
832+
ffsToken string
833+
}
834+
835+
func newFFSHeaderDecorator(ffsToken string) *ffsHeaderDecorator {
836+
return &ffsHeaderDecorator{
837+
ffsToken: ffsToken,
838+
}
839+
}
840+
841+
func (fhd ffsHeaderDecorator) RoundTrip(req *http.Request) (*http.Response, error) {
842+
req.Header["x-ipfs-ffs-auth"] = []string{fhd.ffsToken}
843+
844+
return http.DefaultTransport.RoundTrip(req)
845+
}

‎api/server/server.go‎

Lines changed: 76 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,8 @@ import (
77
"math/big"
88
"net"
99
"net/http"
10+
"net/http/httputil"
11+
"net/url"
1012
"os"
1113
"path/filepath"
1214
"time"
@@ -89,8 +91,9 @@ type Server struct {
8991
grpcServer *grpc.Server
9092
grpcWebProxy *http.Server
9193

92-
gateway *gateway.Gateway
93-
indexServer *http.Server
94+
gateway *gateway.Gateway
95+
indexServer *http.Server
96+
ipfsRevProxy *http.Server
9497

9598
closeLotus func()
9699
}
@@ -111,6 +114,7 @@ type Config struct {
111114
GrpcWebProxyAddress string
112115
RepoPath string
113116
GatewayHostAddr string
117+
IPFSRevProxyAddr string
114118
MaxMindDBFolder string
115119
}
116120

@@ -261,11 +265,70 @@ func NewServer(conf Config) (*Server, error) {
261265

262266
s.indexServer = startIndexHTTPServer(s)
263267

268+
s.ipfsRevProxy, err = startIPFSRevProxy(&conf, ffsManager)
269+
if err != nil {
270+
return nil, fmt.Errorf("starting IPFS reverse proxy: %s", err)
271+
}
272+
264273
log.Info("Starting finished, serving requests")
265274

266275
return s, nil
267276
}
268277

278+
type ffsHTTPAuth struct {
279+
cont http.Handler
280+
ffsManager *manager.Manager
281+
}
282+
283+
func newHTTPFFSAuthInterceptor(cont http.Handler, m *manager.Manager) *ffsHTTPAuth {
284+
fha := &ffsHTTPAuth{
285+
cont: cont,
286+
ffsManager: m,
287+
}
288+
289+
return fha
290+
}
291+
292+
func (fha *ffsHTTPAuth) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
293+
authFFS := r.Header.Get("x-ipfs-ffs-auth")
294+
_, err := fha.ffsManager.GetByAuthToken(authFFS)
295+
if authFFS == "" || err == manager.ErrAuthTokenNotFound {
296+
http.Error(rw, "FFS token required", http.StatusUnauthorized)
297+
return
298+
}
299+
fha.cont.ServeHTTP(rw, r)
300+
}
301+
302+
func startIPFSRevProxy(conf *Config, m *manager.Manager) (*http.Server, error) {
303+
if conf.IPFSRevProxyAddr == "" {
304+
return nil, nil
305+
}
306+
307+
log.Info("Starting IPFS reverse proxy...")
308+
ipfsIP, err := util.TCPAddrFromMultiAddr(conf.IpfsAPIAddr)
309+
if err != nil {
310+
return nil, fmt.Errorf("converting IPFS multiaddr to tcp addr: %s", err)
311+
}
312+
313+
urlIPFS, err := url.Parse("http://" + ipfsIP)
314+
if err != nil {
315+
return nil, fmt.Errorf("generating IPFS URL for reverse proxy: %s", err)
316+
}
317+
rph := httputil.NewSingleHostReverseProxy(urlIPFS)
318+
fha := newHTTPFFSAuthInterceptor(rph, m)
319+
rp := &http.Server{
320+
Addr: conf.IPFSRevProxyAddr,
321+
Handler: fha,
322+
}
323+
324+
go func() {
325+
if err := rp.ListenAndServe(); err != nil && err != http.ErrServerClosed {
326+
log.Fatalf("serving index http: %v", err)
327+
}
328+
}()
329+
return rp, nil
330+
}
331+
269332
func createGRPCServer(opts []grpc.ServerOption, webProxyAddr string) (*grpc.Server, *http.Server) {
270333
grpcServer := grpc.NewServer(opts...)
271334

@@ -384,17 +447,26 @@ func startIndexHTTPServer(s *Server) *http.Server {
384447

385448
// Close shuts down the server.
386449
func (s *Server) Close() {
450+
if s.ipfsRevProxy != nil {
451+
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
452+
defer cancel()
453+
if err := s.ipfsRevProxy.Shutdown(ctx); err != nil {
454+
log.Errorf("closing down ipfs reverse proxy: %s", err)
455+
}
456+
}
457+
387458
ctx, cancel := context.WithTimeout(context.Background(), time.Second)
388459
defer cancel()
460+
389461
if err := s.indexServer.Shutdown(ctx); err != nil {
390-
log.Errorf("shutting down index server: %s", err)
462+
log.Errorf("closing down index server: %s", err)
391463
}
392464

393465
log.Info("closing gRPC endpoints...")
394466
ctx, cancel = context.WithTimeout(context.Background(), time.Second)
395467
defer cancel()
396468
if err := s.grpcWebProxy.Shutdown(ctx); err != nil {
397-
log.Errorf("error shutting down proxy: %s", err)
469+
log.Errorf("closing down proxy: %s", err)
398470
}
399471
stopped := make(chan struct{})
400472
go func() {

‎cli-docs/pow/pow_ffs.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,7 +36,7 @@ Provides commands to manage ffs
3636
* [pow ffs retrievals](pow_ffs_retrievals.md) - List retrieval deal records for an FFS instance
3737
* [pow ffs send](pow_ffs_send.md) - Send fil from one managed address to any other address
3838
* [pow ffs show](pow_ffs_show.md) - Show pinned cid data
39-
* [pow ffs stage](pow_ffs_stage.md) - Temporarily cache data in the Hot layer in preparation for pushing a cid storage config
39+
* [pow ffs stage](pow_ffs_stage.md) - Temporarily stage data in the Hot layer in preparation for pushing a cid storage config
4040
* [pow ffs storage](pow_ffs_storage.md) - List storage deal records for an FFS instance
4141
* [pow ffs watch](pow_ffs_watch.md) - Watch for job status updates
4242

‎cli-docs/pow/pow_ffs_get.md‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,10 @@ pow ffs get [cid] [output file path] [flags]
1313
### Options
1414

1515
```
16-
-h, --help help for get
17-
-t, --token string token of the request
16+
-f, --folder Indicates that the retrieved Cid is a folder
17+
-h, --help help for get
18+
--ipfsrevproxy string Powergate IPFS reverse proxy multiaddr (default "127.0.0.1:6003")
19+
-t, --token string token of the request
1820
```
1921

2022
### Options inherited from parent commands

‎cli-docs/pow/pow_ffs_stage.md‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
## pow ffs stage
22

3-
Temporarily cache data in the Hot layer in preparation for pushing a cid storage config
3+
Temporarily stage data in the Hot layer in preparation for pushing a cid storage config
44

55
### Synopsis
66

7-
Temporarily cache data in the Hot layer in preparation for pushing a cid storage config
7+
Temporarily stage data in the Hot layer in preparation for pushing a cid storage config
88

99
```
1010
pow ffs stage [path] [flags]
@@ -13,8 +13,9 @@ pow ffs stage [path] [flags]
1313
### Options
1414

1515
```
16-
-h, --help help for stage
17-
-t, --token string FFS access token
16+
-h, --help help for stage
17+
--ipfsrevproxy string Powergate IPFS reverse proxy multiaddr (default "127.0.0.1:6003")
18+
-t, --token string FFS access token
1819
```
1920

2021
### Options inherited from parent commands

0 commit comments

Comments
 (0)